Taller and sleeker smartphone devices are becoming the new norm. More screen space and very responsive touchscreens have made for enjoyable experiences available to us at all times. However, after years of interacting with smaller, portable devices, we still try to use these large smartphones on the go, and do not want to change how, where, and when we interact with them. The older devices were easier to use with one hand, when mobile. Now, with bigger devices, users have trouble accessing all parts of the screen with one hand. We need to recognize the limitations in usability due to these large screens. We must start designing user interfaces that are more conducive to one hand usage, which is the preferred way of interacting with the phone. This paper introduces Adaptive App Design, a design methodology that promotes dynamic and adaptive interfaces for one handed usage. We present a novel method of recognizing which hand the user is interacting with and suggest how to design friendlier interfaces for them by presenting a set of design guidelines for this methodology.
Researchers have tried to identify which hand is holding the phone with the help of additional sensors or equipment. @cite_2 @cite_7 @cite_36 @cite_37 @cite_19 used capacitive sensors along the sides of the phone to detect grip while @cite_34 used additional accelerometers. @cite_26 on the other hand used the position of the fingers on the back of the phone to determine the finger interacting with the phone. This method also used additional capacitive sensors.
@cite_37 @cite_26 @cite_7 @cite_36 @cite_19 @cite_2 @cite_34
152129288 2070931068 2402069821 2150639461 1987220739 2765994785 2067689171 2057299504 2737781003 1533974647 2408661706 2113123816 2139916749 2125959037 1983317912 2304267454 1986707552 2124120479 2564823405 2074367177
This paper presents a novel user interface for handheld mobile devices by recognizing hand grip patterns. Particularly, we consider the scenario where the device is provided with an array of capacitive touch sensors underneath the exterior cover. In order to provide the users with intuitive and natural manipulation experience, we use pattern recognition techniques for identifying the users' band grips from the touch sensors. Preliminary user studies suggest that filtering out unintended user hand grip is one of the most important issues to be resolved. We discuss the details of the prototype implementation, as well as engineering challenges for practical deployment. We demonstrate that front-of-screen targeting on mobile phones can be predicted from back-of-device grip manipulations. Using simple, low-resolution capacitive touch sensors placed around a standard phone, we outline a machine learning approach to modelling the grip modulation and inferring front-of-screen touch targets. We experimentally demonstrate that grip is a remarkably good predictor of touch, and we can predict touch position 200ms before contact with an accuracy of 18mm. Previous work on muscle activity sensing has leveraged specialized sensors such as electromyography and force sensitive resistors. While these sensors show great potential for detecting finger hand gestures, they require additional hardware that adds to the cost and user discomfort. Past research has utilized sensors on commercial devices, focusing on recognizing gross hand gestures. In this work we present Serendipity, a new technique for recognizing unremarkable and fine-motor finger gestures using integrated motion sensors (accelerometer and gyroscope) in off-the-shelf smartwatches. Our system demonstrates the potential to distinguish 5 fine-motor gestures like pinching, tapping and rubbing fingers with an average f1-score of 87 . Our work is the first to explore the feasibility of using solely motion sensors on everyday wearable devices to detect fine-grained gestures. This promising technology can be deployed today on current smartwatches and has the potential to be applied to cross-device interactions, or as a tool for research in fields involving finger and hand motion. Mobile phones are increasingly equipped with a range of highly responsive sensors. From cameras and GPS receivers to three-axis accelerometers, applications running on these devices are able to experience rich interactions with their environment. Unfortunately, some applications may be able to use such sensors to monitor their surroundings in unintended ways. In this paper, we demonstrate that an application with access to accelerometer readings on a modern mobile phone can use such information to recover text entered on a nearby keyboard. Note that unlike previous emanation recovery papers, the accelerometers on such devices sample at near the Nyquist rate, making previous techniques unworkable. Our application instead detects and decodes keystrokes by measuring the relative physical position and distance between each vibration. We then match abstracted words against candidate dictionaries and record word recovery rates as high as 80 . In so doing, we demonstrate the potential to recover significant information from the vicinity of a mobile device without gaining access to resources generally considered to be the most likely sources of leakage (e.g., microphone, camera). As mobile and tangible devices are getting smaller and smaller it is desirable to extend the interaction area to their whole surface area. The HandSense prototype employs capacitive sensors for detecting when it is touched or held against a body part. HandSense is also able to detect in which hand the device is held, and how. The general properties of our approach were confirmed by a user study. HandSense was able to correctly classify over 80 percent of all touches, discriminating six different ways of touching the device (hold left right, pick up left right, pick up at top bottom). This information can be used to implement or enhance implicit and explicit interaction with mobile phones and other tangible user interfaces. For example, graphical user interfaces can be adjusted to the user's handedness. A successful grasp requires careful balancing of the contact forces. Deducing whether a particular grasp will be successful from indirect measurements, such as vision, is therefore quite challenging, and direct sensing of contacts through touch sensing provides an appealing avenue toward more successful and consistent robotic grasping. However, in order to fully evaluate the value of touch sensing for grasp outcome prediction, we must understand how touch sensing can influence outcome prediction accuracy when combined with other modalities. Doing so using conventional model-based techniques is exceptionally difficult. In this work, we investigate the question of whether touch sensing aids in predicting grasp outcomes within a multimodal sensing framework that combines vision and touch. To that end, we collected more than 9,000 grasping trials using a two-finger gripper equipped with GelSight high-resolution tactile sensors on each finger, and evaluated visuo-tactile deep neural network models to directly predict grasp outcomes from either modality individually, and from both modalities together. Our experimental results indicate that incorporating tactile readings substantially improve grasping performance. To evaluate our three-axis tactile sensor developed in preceding papers, a tactile sensor is mounted on a robotic finger with 3-degrees of freedom. We develop a dual computer system that possesses two computers to enhance processing speed: one is for tactile information processing and the other controls the robotic finger; these computers are connected to a local area network. Three kinds of experiments are performed to evaluate the robotic finger's basic abilities required for dexterous hands. First, the robotic hand touches and scans flat specimens to evaluate their surface condition. Second, it detects objects with parallelepiped and cylindrical contours. Finally, it manipulates a parallelepiped object put on a table by sliding it. Since the present robotic hand performed the above three tasks, we conclude that it is applicable to the dexterous hand in subsequent studies. We present a novel robotic grasp controller that allows a sensorized parallel jaw gripper to gently pick up and set down unknown objects once a grasp location has been selected. Our approach is inspired by the control scheme that humans employ for such actions, which is known to centrally depend on tactile sensation rather than vision or proprioception. Our controller processes measurements from the gripper's fingertip pressure arrays and hand-mounted accelerometer in real time to generate robotic tactile signals that are designed to mimic human SA-I, FA-I, and FA-II channels. These signals are combined into tactile event cues that drive the transitions between six discrete states in the grasp controller: Close, Load, Lift and Hold, Replace, Unload, and Open. The controller selects an appropriate initial grasping force, detects when an object is slipping from the grasp, increases the grasp force as needed, and judges when to release an object to set it down. We demonstrate the promise of our approach through implementation on the PR2 robotic platform, including grasp testing on a large number of real-world objects. In-hand manipulation is certainly one of the most challenging problems in robotic manipulation. Solutions to this problem depend on the specific device used to grab the object, but nowadays, the trend is to exploit not only the gripper but also external constraints, such as other objects in the environment or external forces, like gravity. This allows a robot to manipulate an object even with very simple grippers, like a parallel gripper. Nevertheless, even for a simple grasping task, which aims at grabbing the object with a given fixed orientation or for executing a controlled slip, information on the contact between the fingers of the gripper and the object is relevant. In these cases, both linear and rotational slipping should be controlled during the grasping phase and during the motion phase. The present paper proposes a control strategy for the first objective, namely slipping avoidance. The strategy is based on contact information provided by a six-axis force tactile sensor, able to measure contact force and torque as well as able to provide information on the contact geometry, that means orientation of the object with respect to the gripper. Experiments on a parallel gripper sensorized with a new force tactile sensor and mounted on a Kuka iiwa show how the strategy successfully allows the robot to safely manipulate a rigid object in various friction conditions of its surface. Recent researches have shown that motion sensors may be used as a side channel to infer keystrokes on the touchscreen of smartphones. However, the practicality of this attack is unclear. For example, does this attack work on different devices, screen dimensions, keyboard layouts, or keyboard types? Does this attack depend on specific users or is it user independent? To answer these questions, we conducted a user study where 21 participants typed a total of 47,814 keystrokes on four different mobile devices in six settings. Our results show that this attack remains effective even though the accuracy is affected by user habits, device dimension, screen orientation, and keyboard layout. On a number-only keyboard, after the attacker tries 81 4-digit PINs, the probability that she has guessed the correct PIN is 65 , which improves the accuracy rate of random guessing by 81 times. Our study also indicates that inference based on the gyroscope is more accurate than that based on the accelerometer. We evaluated two classification techniques in our prototype and found that they are similarly effective. The proliferation of wearable devices, e.g., smartwatches and activity trackers, with embedded sensors has already shown its great potential on monitoring and inferring human daily activities. This paper reveals a serious security breach of wearable devices in the context of divulging secret information (i.e., key entries) while people accessing key-based security systems. Existing methods of obtaining such secret information relies on installations of dedicated hardware (e.g., video camera or fake keypad), or training with labeled data from body sensors, which restrict use cases in practical adversary scenarios. In this work, we show that a wearable device can be exploited to discriminate mm-level distances and directions of the user's fine-grained hand movements, which enable attackers to reproduce the trajectories of the user's hand and further to recover the secret key entries. In particular, our system confirms the possibility of using embedded sensors in wearable devices, i.e., accelerometers, gyroscopes, and magnetometers, to derive the moving distance of the user's hand between consecutive key entries regardless of the pose of the hand. Our Backward PIN-Sequence Inference algorithm exploits the inherent physical constraints between key entries to infer the complete user key entry sequence. Extensive experiments are conducted with over 5000 key entry traces collected from 20 adults for key-based security systems (i.e. ATM keypads and regular keyboards) through testing on different kinds of wearables. Results demonstrate that such a technique can achieve 80 accuracy with only one try and more than 90 accuracy with three tries, which to our knowledge, is the first technique that reveals personal PINs leveraging wearable devices without the need for labeled training data and contextual information. This paper reports on the development of a hand to machine interface device that provides real-time gesture, position and orientation information. The key element is a glove and the device as a whole incorporates a collection of technologies. Analog flex sensors on the glove measure finger bending. Hand position and orientation are measured either by ultrasonics, providing five degrees of freedom, or magnetic flux sensors, which provide six degrees of freedom. Piezoceramic benders provide the wearer of the glove with tactile feedback. These sensors are mounted on the light-weight glove and connected to the driving hardware via a small cable. Applications of the glove and its component technologies include its use in conjunction with a host computer which drives a real-time 3-dimensional model of the hand allowing the glove wearer to manipulate computer-generated objects as if they were real, interpretation of finger-spelling, evaluation of hand impairment in addition to providing an interface to a visual programming language. Gesture recognition has been a research focus with the popularity of depth sensing device. In this paper, we propose a new fingertip detection method based on a novel definition of fingers. This method consists of two steps. Firstly, finger bases are detected and estimated as prior information. Secondly, the finger regions and fingertips are located. In the second module, the point cloud of hand is represented as a graph to obtain all geodesic paths originated from palm center. If one path travels through a finger base, then its terminal point is defined as a finger point. The fingertips are determined within these finger points by utilizing geodesic distances. To our knowledge, such definition has never been applied in gesture recognition before and its performance surpasses the definition of geodesic maxima. Experimental results demonstrates the effectiveness of our method even when hand is not parallel to camera. Compared with state-of-the-art approach, our method shows much less error. It is believed that the use of haptic sensors to measure the magnitude, direction, and distribution of a force will enable a robotic hand to perform dexterous operations. Therefore, we develop a new type of finger-shaped haptic sensor using GelForce technology. GelForce is a vision-based sensor that can be used to measure the distribution of force vectors, or surface traction fields. The simple structure of the GelForce enables us to develop a compact finger-shaped GelForce for the robotic hand. GelForce that is developed on the basis of an elastic theory can be used to calculate surface traction fields using a conversion equation. However, this conversion equation cannot be analytically solved when the elastic body of the sensor has a complicated shape such as the shape of a finger. Therefore, we propose an observational method and construct a prototype of the finger-shaped GelForce. By using this prototype, we evaluate the basic performance of the finger-shaped GelForce. Then, we conduct a field test by performing grasping operations using a robotic hand. The results of this test show that using the observational method, the finger-shaped GelForce can be successfully used in a robotic hand. Three studies of different mobile-device hand postures are presented. The first study measures the performance of postures in Fitts' law tasks using one and two hands, thumbs and index fingers, horizontal and vertical movements, and front- and back-of-device interaction. Results indicate that the index finger performs well on both the front and the back of the device, and that thumb performance on the front of the device is generally worse. Fitts' law models are created and serve as a basis for comparisons. The second study examines the orientation of shapes on the front and back of a mobile device. It shows that participants' expectations of visual feedback for finger movements on the back of a device reverse the direction of their finger movements to favor a ''transparent device'' orientation. The third study examines letter-like gestures made on the front and back of a device. It confirms the performance of the index finger on the front of the device, while showing limitations in the ability for the index finger on the back to perform complex gestures. Taken together, these results provide an empirical foundation upon which new mobile interaction designs can be based. A set of design implications and recommendations are given based directly on the findings presented. The position of on-body motion sensors plays an important role in human activity recognition. Most often, mobile phone sensors at the trouser pocket or an equivalent position are used for this purpose. However, this position is not suitable for recognizing activities that involve hand gestures, such as smoking, eating, drinking coffee and giving a talk. To recognize such activities, wrist-worn motion sensors are used. However, these two positions are mainly used in isolation. To use richer context information, we evaluate three motion sensors (accelerometer, gyroscope and linear acceleration sensor) at both wrist and pocket positions. Using three classifiers, we show that the combination of these two positions outperforms the wrist position alone, mainly at smaller segmentation windows. Another problem is that less-repetitive activities, such as smoking, eating, giving a talk and drinking coffee, cannot be recognized easily at smaller segmentation windows unlike repetitive activities, like walking, jogging and biking. For this purpose, we evaluate the effect of seven window sizes (2–30 s) on thirteen activities and show how increasing window size affects these various activities in different ways. We also propose various optimizations to further improve the recognition of these activities. For reproducibility, we make our dataset publicly available. Smartwatch is becoming one of the most popular wearable device with many major smartphone manufacturers such as Samsung and Apple releasing their smartwatches recently. Apart from the fitness applications, the smartwatch provides a rich user interface that has enabled many applications like instant messaging and email. Since the smartwatch is worn on the wrist, it introduces a unique opportunity to understand user's arm, hand and possibly finger movements using its accelerometer and gyroscope sensors. Although user's arm and hand gestures are likely to be identified with ease using the smartwatch sensors, it is not clear how much of user's finger gestures can be recognized. In this paper, we show that motion energy measured at the smartwatch is sufficient to uniquely identify user's hand and finger gestures. We identify essential features of accelerometer and gyroscope data that reflect the movements of tendons (passing through the wrist) when performing a finger or a hand gesture. With these features, we build a classifier that can uniquely identify 37 (13 finger, 14 hand and 10 arm) gestures with an accuracy of 98 . We further extend our gesture recognition to identify the characters written by the user with her index finger on a surface, and show that such finger-writing can also be accurately recognized with nearly 95 accuracy. Our presented results will enable many novel applications like remote control and finger-writing-based input to devices using smartwatch. An experiment was conducted to investigate gesture recognition with a human hand manipulating the DataGlove, an electronically instrumented glove which provides information about finger and hand position. A total of 22 gestures in three classes were investigated. The first class contained gestures which only involved finger flexure. The second class contained gestures which required both finger flexure and hand orientation. The third class of gestures required finger motion in addition to flexure and orientation. Only four sensors were necessary to positively identify specific gestures from groups of up to 15 gestures. The results show the specific number of sensors required to positively identify a gesture from a group. This depends on the number of gestures in a group, as well as the class of gestures. > Abstract This paper describes the development of a remote handling control of an anthropomorphic robotic hand, the SCHUNK S5FH, using the human hand as master by measuring its motion with OptiTrack Technology. The goal of this work is to enhance manipulation studies on the human hand and to instantly transfer those studies on robotic hands. A preliminary study on methods and devices used for fingers tracking led to the choice of a simplified kinematic model of the human hand on the basis of the available motion tracking system. Using the same criteria, the analysis of protocols for markers allocation led to define the number and a method for their arrangement on the fingers and palm. In order to overcome the limitation of the Motion Capture System, a method for identification and labeling has been developed according to their anatomical arrangement. Afterwards, the tracking is performed using the constraints between marker positions on the kinematic chain of the hand and a dynamic labeling algorithm robust with respect to noise, outliers and loss of markers. The validation is performed using the right hand of different subjects and considering different tasks involving flexion extension and abduction adduction of fingers and thumb opposition. For testing and validation, preliminary studies on synergies for manipulation tasks such as screwing a cup, has been conducted on the human hand and transferred on the robotic hand. We show that accelerometer readings are a powerful side channel that can be used to extract entire sequences of entered text on a smart-phone touchscreen keyboard. This possibility is a concern for two main reasons. First, unauthorized access to one's keystrokes is a serious invasion of privacy as consumers increasingly use smartphones for sensitive transactions. Second, unlike many other sensors found on smartphones, the accelerometer does not require special privileges to access on current smartphone OSes. We show that accelerometer measurements can be used to extract 6-character passwords in as few as 4.5 trials (median).