Hand segmentation and fingertip detection play an indispensable role in hand gesture-based human-machine interaction systems. In this study, we propose a method to discriminate hand components and to locate fingertips in RGB-D images. The system consists of three main steps: hand detection using RGB images providing regions which are considered as promising areas for further processing, hand segmentation, and fingertip detection using depth image and our modified SegNet, a single lightweight architecture that can process two independent tasks at the same time. The experimental results show that our system is a promising method for hand segmentation and fingertip detection which achieves a comparable performance while model complexity is suitable for real-time applications.
To address the fingertip detection issue, researchers have clearly focused on two main approaches in terms of input signals: the first only uses RGB images while the other considers depth images as well. The latter consistently acquires a reputation for its performance compared to the former since it uses both RGB values and depth signals from the camera. Regarding the first approach, researchers have proposed hand image processing methods based on background subtraction and skin color detection @cite_2 @cite_9 . However, this approach can be affected by the illumination and variations in the background of the sampling environment, that is, the authors assumed that the images were captured from stationary cameras in a steady scenario. The benefit of this hypothesis is that such simple computer vision techniques can be employed as background subtraction, etc. The other approach for hand image processing, which is to use RGB-D images, was applied in @cite_1 @cite_6 . In lieu of using a hand detector for RGB images, the authors located fingertips by assuming that the hand must be at the shortest distance from the camera with the given depth information.
@cite_9 @cite_1 @cite_6 @cite_2
2513258067 2005255079 1970629627 2776256943
With the heated trend of augmented reality (AR) and popularity of smart head-mounted devices, the development of natural human device interaction is important, especially the hand gesture based interaction. This paper presents a solution for the point gesture based interaction in the egocentric vision and its application. Firstly, a dataset named EgoFinger is established focusing on the pointing gesture for the egocentric vision. We discuss the dataset collection detail and as well the comprehensive analysis of this dataset, including background and foreground color distribution, hand occurrence likelihood, scale and pointing angle distribution of hand and finger, and the manual labeling error analysis. The analysis shows that the dataset covers substantial data samples in various environments and dynamic hand shapes. Furthermore, we propose a two-stage Faster R-CNN based hand detection and dual-target fingertip detection framework. Comparing with state-of-art tracking and detection algorithm, it performs the best in both hand and fingertip detection. With the large-scale dataset, we achieve fingertip detection error at about 12.22 pixels in 640px × 480px video frame. Finally, using the fingertip detection result, we design and implement an input system for the egocentric vision, i.e., Ego-Air-Writing. By considering the fingertip as a pen, the user with wearable glass can write character in the air and interact with system using simple hand gestures. We propose a real-time finger writing character recognition system using depth information. This system allows user to input characters by writing freely in the air with the Kinect. During the writing process, it is reasonable to assume that the finger and hand are always holding in front of torso. Firstly, we compute the depth histogram of human body and use a switch mixture Gaussian model to characterize it. Since the hand is closer to camera, a model-based threshold can segment the hand-related region out. Then, we employ an unsupervised clustering algorithm, K-means, to classify the segmented region into two parts, the finger-hand part and hand-arm part. By identifying the arm direction, we can determine the finger-hand cluster and locate the fingertip as the farthest point from the other cluster. We collected over 8000 frames writing-in-the-air sequences including two different subjects writing numbers, strokes, pattern, English and Chinese characters from two different distances. From our experiments, the proposed algorithm can provide robust and accurate fingertip detection, and achieve encouraging character recognition result. Egocentric cameras can be used to benefit such tasks as analyzing fine motor skills, recognizing gestures and learning about hand-object manipulation. To enable such technology, we believe that the hands must detected on the pixel-level to gain important information about the shape of the hands and fingers. We show that the problem of pixel-wise hand detection can be effectively solved, by posing the problem as a model recommendation task. As such, the goal of a recommendation system is to recommend the n-best hand detectors based on the probe set - a small amount of labeled data from the test distribution. This requirement of a probe set is a serious limitation in many applications, such as ego-centric hand detection, where the test distribution may be continually changing. To address this limitation, we propose the use of virtual probes which can be automatically extracted from the test distribution. The key idea is that many features, such as the color distribution or relative performance between two detectors, can be used as a proxy to the probe set. In our experiments we show that the recommendation paradigm is well-equipped to handle complex changes in the appearance of the hands in first-person vision. In particular, we show how our system is able to generalize to new scenarios by testing our model across multiple users. The complementary nature of color and depth synchronized information acquired by low cost RGBD sensors poses new challenges and design opportunities in several applications and research areas. Here, we focus on background subtraction for moving object detection, which is the building block for many computer vision applications, being the first relevant step for subsequent recognition, classification, and activity analysis tasks. The aim of this paper is to describe a novel benchmarking framework that we set up and made publicly available in order to evaluate and compare scene background modeling methods for moving object detection on RGBD videos. The proposed framework involves the largest RGBD video dataset ever made for this specific purpose. The 33 videos span seven categories, selected to include diverse scene background modeling challenges for moving object detection. Seven evaluation metrics, chosen among the most widely used, are adopted to evaluate the results against a wide set of pixel-wise ground truths. Moreover, we present a preliminary analysis of results, devoted to assess to what extent the various background modeling challenges pose troubles to background subtraction methods exploiting color and depth information.