Abstract Designing of touchless user interface is gaining popularity in various contexts. Users can interact with electronic devices using such interfaces even when their hands are dirty or non-conductive. Also, users with partial physical disability can interact with electronic devices with the help of touchless interfaces. In this paper, we propose a Leap Motion controller-based methodology to facilitate rendering of 2D and 3D shapes on display devices. The proposed method tracks finger movements while users perform natural gestures within the field of view of the motion sensor. Then, trajectories are analyzed to extract extended Npen++ features in 3D. These features capture finger movements during the gestures and they are fed to unidirectional left-to-right Hidden Markov Model (HMM) for training. A one-to-one mapping between gestures and shapes, is proposed. Finally, the shapes corresponding to these gestures are rendered over the display using a typical MuPad supported interface. We have created a dataset of 5400 samples recorded by 10 volunteers. Our dataset contains 18 geometric and 18 non-geometric shapes such as “circle”, “rectangle”, “flower”, “cone”, “sphere”, etc. The proposed method has achieved 92.87 accuracy using a 5-fold cross validation scheme. Experiments reveal that the extended 3D features perform better than the existing 3D features when applied for shape representation and classification. The method can be used for developing diverse HCI applications suitable for smart display devices.
Despite good accuracy of the sensor-based systems, they have certain drawbacks; (i) Contact-based systems are a burden to the users because often they feel uncomfortable with such artificial attachments @cite_7 (ii) Some of the existing systems require external power through battery @cite_22 . Therefore, contact-less vision-guided systems are preferred for such applications. Researchers have shown that Leap Motion device can be successfully used for palm rehabilitation @cite_3 , upper limb rehabilitation @cite_47 , stroke rehabilitation @cite_6 etc.
@cite_22 @cite_7 @cite_3 @cite_6 @cite_47
2339176273 2160770229 2162522697 2059812677
Physical rehabilitation supported by the computer-assisted-interface is gaining popularity among health-care fraternity. In this paper, we have proposed a computer-vision-assisted contactless methodology to facilitate palm and finger rehabilitation. Leap motion controller has been interfaced with a computing device to record parameters describing 3-D movements of the palm of a user undergoing rehabilitation. We have proposed an interface using Unity3D development platform. Our interface is capable of analyzing intermediate steps of rehabilitation without the help of an expert, and it can provide online feedback to the user. Isolated gestures are classified using linear discriminant analysis (DA) and support vector machines (SVM). Finally, a set of discrete hidden Markov models (HMM) have been used to classify gesture sequence performed during rehabilitation. Experimental validation using a large number of samples collected from healthy volunteers reveals that DA and SVM perform similarly while applied on isolated gesture recognition. We have compared the results of HMM-based sequence classification with CRF-based techniques. Our results confirm that both HMM and CRF perform quite similarly when tested on gesture sequences. The proposed system can be used for home-based palm or finger rehabilitation in the absence of experts. In recent years, the field of Human-Computer Interaction (HCI) has been advanced with many technologies, however, most are limited to healthy users. In this paper, we leveraged the technology of free-hand interaction to rehabilitate patients with stroke. We modified the game of Fruit Ninja to use Leap Motion controller's hand tracking data for stroke patients with arm and hand weakness to practice their finger individuation. In a pilot study, we recruited 14 patients with chronic stroke to play the game using natural interaction. Their Fruit Ninja (FN) scores show high correlation with the standard clinical assessment scores such as Fugl-Meyer (FMA) and Box-and-Blocks Test (BBT) scores. This finding suggests that our free-hand Fruit Ninja's score is a good indicator of the patient's hand function and therefore will be informative if used in their rehabilitation. The complexity of hand function is such that most existing upper limb rehabilitation robotic devices use only simplified hand interfaces. This is in contrast to the importance of the hand in regaining function after neurological injury. Computer vision technology has been used to identify hand posture in the field of Human Computer Interaction, but this approach has not been translated to the rehabilitation context. We describe a computer vision-based classifier that can be used to discriminate rehabilitation-relevant hand postures, and could be integrated into a virtual reality-based upper limb rehabilitation system. The proposed system was tested on a set of video recordings from able-bodied individuals performing cylindrical grasps, lateral key grips, and tip-to-tip pinches. The overall classification success rate was 91.2 , and was above 98 for 6 out of the 10 subjects. Monitoring and interpreting (sub)cortical reorganization after stroke may be useful for selecting therapies and improving rehabilitation outcome. To develop computational models that predict behavioral motor improvement from changing brain activation pattern, we are currently working on the implementation of a clinically feasible experimental set-up, which enables recording high quality electroencephalography (EEG) signals during inpatient rehabilitation of upper and lower limbs. The major drawback of current experimental paradigms is the cue-guided repetitive design and the lack of functional movements. In this paper, we assess the usability of the Kinect device (Microsoft Inc., Redmond, WA, USA) for tracking self-paced hand opening and closing movements. Three able-bodied volunteers performed self-paced right hand open-close movement sequences while EEG was recorded from sensorimotor areas and electromyography (EMG) from the right arm from extensor carpi radialis and flexor carpi radialis muscles. The results of the study suggest that the Kinect device allows generation of trigger information that is comparable to the information that can be obtained from EMG.