Apple Watch can learn new hand gestures from one minute of practice

An Apple Watch can be taught to recognize new hand motions after about one minute of practice per motion. Researchers from Northwestern University and Carnegie Mellon University built a system called TransfHAR that reached 90% accuracy on seven new activities. It is a research project, and no release has been announced.

What the researchers did

Today the Apple Watch recognizes only two gestures: a two-finger tap and a wrist flick. TransfHAR uses two sensors already inside the watch, an accelerometer and a gyroscope, which track how your wrist moves and turns. The system first learned from about 2,760 hours of public wrist movement data from 381 people, mostly broad activities like sitting, walking and exercising.

Then 10 participants wore an Apple Watch Series 10. Each picked seven everyday activities, such as stirring coffee, drawing a heart in the air, applying hand sanitizer, opening a Gatorade bottle, tying a shoelace and filing nails.

How well it worked

With one minute of recorded movement per activity, the watch identified the seven new motions with 90% accuracy. Even less practice helped. The figures below are averages across all the activities.

• One 2.56-second clip per activity: 75.2% accuracy
• Five clips, about 13 seconds: 86.7%
• Twenty clips, about 51 seconds: 89.3%
• Pooling 70 activities in one model: still above 82%

All of this ran on the watch itself. Learning a new set of activities took less than a second, and a prediction took about 24 milliseconds. No paired phone or remote server was needed.

What it could be used for

The idea is that a motion could start an action with no tapping or talking. A coffee-stirring motion could play music or open a caffeine tracker. A bicep curl could start logging a workout. The researchers also point to medical uses, like passively measuring tremors, and to factory systems that track assembly steps. Smart home controls are another possibility.

One team member is a research scientist from Google, and the core model is adapted from a Google wearable sensor model called LSM-2. That hints the approach could reach other smartwatches, since most collect the same kind of motion data.

The limits

The system confused activities that involve small, steady hand movements, such as writing, scrolling on a phone and applying lotion. To the sensors, these look alike. It also had a harder time with multi-step routines, like the separate steps of making a latte.

The study was short and controlled, and the team did not measure the effect on battery life. Training tools are not publicly available yet, so it is unclear when anyone could try this on their own watch.