As mobile and wearable devices such as smartwatches grow smaller, it gets harder and harder for people to interact with screens the size of a matchbook.
But that may change. New sonar technology lets you interact with mobile devices by writing or gesturing on any nearby surface—a tabletop, a sheet of paper, or even in mid-air.
The technology, FingerIO, tracks fine-grained finger movements by turning a smartphone or smartwatch into an active sonar system using the device’s own microphones and speakers.
Because sound waves travel through fabric and do not require a line of sight, users can even interact with a phone inside a pocket or a smartwatch hidden under a sleeve.
In a paper that will be presented in May at the Association for Computing Machinery’s CHI 2016 conference, researchers show that FingerIO accurately tracks two-dimensional finger movements to within 8mm, sufficiently accurate to interact with mobile devices.
“You can’t type very easily onto a smartwatch display, so we wanted to transform a desk or any area around a device into an input surface,” says lead author Rajalakshmi Nandakumar, a doctoral student in computer science and engineering at the University of Washington.
“I don’t need to instrument my fingers with any other sensors—I just use my finger to write something on a desk or any other surface and the device can track it with high resolution.”
Using FingerIO, one could use the flick of a finger to turn up the volume, press a button, or scroll through menus on a smartphone without touching it, or even write a search command or text in the air rather than typing on a tiny screen.
The device turns a smartwatch or smartphone into a sonar system using the device’s own speaker to emit an inaudible sound wave. That signal bounces off the finger, and those “echoes” are recorded by the device’s microphones and used to calculate the finger’s location in space.
Using sound waves to track finger motion offers several advantages over cameras—which don’t work without line-of-sight when the device is hidden by fabric or another obstructions—and other technologies like radar that require both custom sensor hardware and greater computing power, says senior author Shyam Gollakota, assistant professor of computer science and engineering.
“Acoustic signals are great—because sound waves travel much slower than the radio waves used in radar, you don’t need as much processing bandwidth so everything is simpler. And from a cost perspective, almost every device has a speaker and microphones so you can achieve this without any special hardware.”
But sonar echoes are weak and typically not accurate enough to track finger motion at a high resolution. Errors of a few centimeters make it impossible to differentiate between writing individual letters or subtle hand gestures.
Researchers employed a type of signal typically used in wireless communication—called Orthogonal Frequency Division Multiplexing—and demonstrated that it can be used to achieve high-resolution finger tracking using sound. Their algorithms leverage the properties of OFDM signals to track phase changes in the echoes and correct for any errors in the finger location to achieve sub-centimeter finger tracking.
To test the approach, the researchers created a FingerIO prototype app for Android devices and downloaded it to an off-the-shelf Samsung Galaxy S4 smartphone and a smartwatch customized with two microphones, which are needed to track finger motion in two dimensions. Today’s smartwatches typically only have one, which can be used to track a finger in one dimension.
Testers drew shapes such as stars, squiggles, or figure 8s on a touchpad next to a smartphone or smartwatch running FingerIO. They then compared the touchpad tracings to the shapes created by FingerIO’s tracking. The average difference between the drawings and the FingerIO tracings was 0.8 centimeters for the smartphone and 1.2 centimeters for the smartwatch.
“Given that your finger is already a centimeter thick, that’s sufficient to accurately interact with the devices,” says coauthor and electri