Abstract Designing of touchless user interface is gaining popularity in various contexts. Users can interact with electronic devices using such interfaces even when their hands are dirty or non-conductive. Also, users with partial physical disability can interact with electronic devices with the help of touchless interfaces. In this paper, we propose a Leap Motion controller-based methodology to facilitate rendering of 2D and 3D shapes on display devices. The proposed method tracks finger movements while users perform natural gestures within the field of view of the motion sensor. Then, trajectories are analyzed to extract extended Npen++ features in 3D. These features capture finger movements during the gestures and they are fed to unidirectional left-to-right Hidden Markov Model (HMM) for training. A one-to-one mapping between gestures and shapes, is proposed. Finally, the shapes corresponding to these gestures are rendered over the display using a typical MuPad supported interface. We have created a dataset of 5400 samples recorded by 10 volunteers. Our dataset contains 18 geometric and 18 non-geometric shapes such as “circle”, “rectangle”, “flower”, “cone”, “sphere”, etc. The proposed method has achieved 92.87 accuracy using a 5-fold cross validation scheme. Experiments reveal that the extended 3D features perform better than the existing 3D features when applied for shape representation and classification. The method can be used for developing diverse HCI applications suitable for smart display devices.
Lately, gesture based 3D shape creation and recognition has been a topic of research. The VIDEODESK system proposed by @cite_29 can be considered as the pioneering work in this field. They have designed a system that allows a user to control an object's shape by using his her own hands. Whereas utilization of both hands of the user is of good advantage, however, their system uses predefined points of a user's hands. Thus, the system fails to take advantage of full expressive power of both hands. Researchers have shown that, the accuracy can be improved using both hands @cite_41 . Though the improved framework can assign different responsibility to each hand and the method suggests to use bi-manual actions than Krueger's method in the context of 3D shape modeling, the object deformation is only controlled by position and orientation of both hands. Therefore, shape of the hand is not used in true sense. This has been improved by @cite_2 . They have shown that, representation of 3D objects can be improved through bi-manual actions. However, above mentioned methodologies assume that, palm including fingers are detected and tracked precisely for gesture recognition to be effective.
@cite_41 @cite_29 @cite_2
2008110769 2467634805 2112480082 1530957309
In this paper we propose a hand gesture recognition system that relies on color and depth images, and on a small pose sensor on the human palm. Monocular and stereo vision systems have been used for human pose and gesture recognition, but with limited scope due to limitations on texture, illumination, etc. New RGB-Depth sensors, that reply on projected light such as the Microsoft Kinect, have overcome many of those limitations. However, the point clouds for hand gestures are still in many cases noisy and partially occluded, and hand gesture recognition is not trivial. Hand gesture recognition is much more complex than full body motion, since we can have the hands in any orientation and can not assume a standing body on a ground plane. In this work we propose to add a tiny pose sensor to the human palm, with a minute accelerometer and magnetometer that combined provide 3D angular pose, to reduce the search space and have a robust and computationally light recognition method. Starting with the full depth image point cloud, segmentation can be performed by taking into account the relative depth and hand orientation, as well as skin color. Identification is then performed by matching 3D voxel occupancy against a gesture template database. Preliminary results are presented for the recognition of Portuguese Sign Language alphabet, showing the validity of the approach. In this paper, a new skeleton-based approach is proposed for 3D hand gesture recognition. Specifically, we exploit the geometric shape of the hand to extract an effective descriptor from hand skeleton connected joints returned by the Intel RealSense depth camera. Each descriptor is then encoded by a Fisher Vector representation obtained using a Gaussian Mixture Model. A multi-level representation of Fisher Vectors and other skeleton-based geometric features is guaranteed by a temporal pyramid to obtain the final feature vector, used later to achieve the classification by a linear SVM classifier. The proposed approach is evaluated on a challenging hand gesture dataset containing 14 gestures, performed by 20 participants performing the same gesture with two different numbers of fingers. Experimental results show that our skeleton-based approach consistently achieves superior performance over a depth-based approach. For one-shot learning gesture recognition, two important challenges are: how to extract distinctive features and how to learn a discriminative model from only one training sample per gesture class. For feature extraction, a new spatio-temporal feature representation called 3D enhanced motion scale-invariant feature transform (3D EMoSIFT) is proposed, which fuses RGB-D data. Compared with other features, the new feature set is invariant to scale and rotation, and has more compact and richer visual representations. For learning a discriminative model, all features extracted from training samples are clustered with the k-means algorithm to learn a visual codebook. Then, unlike the traditional bag of feature (BoF) models using vector quantization (VQ) to map each feature into a certain visual codeword, a sparse coding method named simulation orthogonal matching pursuit (SOMP) is applied and thus each feature can be represented by some linear combination of a small number of codewords. Compared with VQ, SOMP leads to a much lower reconstruction error and achieves better performance. The proposed approach has been evaluated on ChaLearn gesture database and the result has been ranked amongst the top best performing techniques on ChaLearn gesture challenge (round 2). While many sophisticated 3D rendering methods become available to produce realistic output, 3D data input still is a tedious and time-consuming task. The paper proposes a method for creating 3D objects using hand gestures. Two types of 3D shape forming tasks are considered. One is to create a completely new object from scratch and the other is to make it by changing an existing one. In the first case, a 3D modeling method is designed and implemented to provide functions like creation, blending and deformation of multiple 3D shapes easily controlled by intuitive bimanual gestures. The implicit superquadric functions are applied to building a gesture-based 3D object modeler with extensions of blending and deformations. In the second case, a 3D data management function is proposed to share and reuse existing shapes created by using the 3D modeler mentioned above. All 3D data are kept small in size and retrieved by picturing partial shapes with the user's hands. As a consequence, the proposed method allows users to consistently deal with all object modeling tasks including creation, storage and reuse of 3D objects through natural and intuitive hand gestures. Some 3D objects and a miniature solid model are shown to present the expressive power and usefulness of the proposed method.