Tracking the full skelet al pose of the hands and fingers is a challenging problem that has a plethora of applications for user interaction. Existing techniques either require wearable hardware, add restrictions to user pose, or require significant computation resources. This research explores a new approach to tracking hands, or any articulated model, by using an augmented rigid body simulation. This allows us to phrase 3D object tracking as a linear complementarity problem with a well-defined solution. Based on a depth sensor's samples, the system generates constraints that limit motion orthogonal to the rigid body model's surface. These constraints, along with prior motion, collision contact constraints, and joint mechanics, are resolved with a projected Gauss-Seidel solver. Due to camera noise properties and attachment errors, the numerous surface constraints are impulse capped to avoid overpowering mechanical constraints. To improve tracking accuracy, multiple simulations are spawned at each frame and fed a variety of heuristics, constraints and poses. A 3D error metric selects the best-fit simulation, helping the system handle challenging hand motions. Such an approach enables real-time, robust, and accurate 3D skelet al tracking of a user's hand on a variety of depth cameras, while only utilizing a single x86 CPU core for processing.
By leveraging depth cameras, and recursive state tracking, sequential Monte Carlo methods can search a large hand pose state-space for valid solutions @cite_22 . More recently, a particle swarm approach has demonstrated the ability to fully track a temporally-coherent 3D skelet al hand model @cite_12 . Many sample configurations are rendered to depth buffers, which are compared to the sensor's depth data, enabling iteration toward an error minimizing pose. In contrast to these approaches, which are very compute intensive, our approach comfortably runs over 60 Hz on a single core of an x86 processor. At each frame we move directly toward the best local fit (pose of minimum error) using proven techniques for rigid body simulation, rather than rendering-based trial and error. Robustness and recovering from loss-of-tracking is a significant challenge for any markerless system given the large high dimensional space of hand poses. To increase the reliability that the system predicts the correct pose, and avoid loss of tracking, we need only spawn a few different simulations rather than testing hundreds of potential candidate poses. Furthermore, the system is able to incorporate information, when available, such as as feature identification or known pose information acquired through other methods.
@cite_22 @cite_12
2168392347 2100642335
Gesture recognition pertains to recognizing meaningful expressions of motion by a human, involving the hands, arms, face, head, and or body. It is of utmost importance in designing an intelligent and efficient human-computer interface. The applications of gesture recognition are manifold, ranging from sign language through medical rehabilitation to virtual reality. In this paper, we provide a survey on gesture recognition with particular emphasis on hand gestures and facial expressions. Applications involving hidden Markov models, particle filtering and condensation, finite-state machines, optical flow, skin color, and connectionist models are discussed in detail. Existing challenges and future research possibilities are also highlighted We present a novel solution to the problem of recovering and tracking the 3D position, orientation and full articulation of a human hand from markerless visual observations obtained by a Kinect sensor. We treat this as an optimization problem, seeking for the hand model parameters that minimize the discrepancy between the appearance and 3D structure of hypothesized instances of a hand model and actual hand observations. This optimization problem is effectively solved using a variant of Particle Swarm Optimization (PSO). The proposed method does not require special markers and or a complex image acquisition setup. Being model based, it provides continuous solutions to the problem of tracking hand articulations. Extensive experiments with a prototype GPU-based implementation of the proposed method demonstrate that accurate and robust 3D tracking of hand articulations can be achieved in near real-time (15Hz).