Recovery of articulated 3D structure from 2D observations is a challenging computer vision problem with many applications. Current learning-based approaches achieve state-of-the-art performance on public benchmarks but are limited to the specific types of objects and motions covered by the training datasets. Model-based approaches do not rely on training data but show lower accuracy on public benchmarks. In this paper, we introduce a new model-based method called Structure from Articulated Motion (SfAM). SfAM includes a new articulated structure term which ensures consistency of bone lengths throughout the whole image sequence and recovers a scene-specific configuration of the articulated structure. The proposed approach is highly robust to noisy 2D annotations, generalizes to arbitrary objects and motion types and does not rely on training data. It achieves state-of-the-art accuracy and scales across different scenarios which is shown in extensive experiments on public benchmarks and real video sequences.
Recently, many learning-based approaches for human pose and hand pose estimation have been presented in the literature @cite_37 @cite_31 @cite_81 @cite_26 @cite_4 @cite_16 @cite_39 . These methods are highly specialized and rely on large collections of training data. In contrast, our SfAM is a general approach which can cope with different articulated structures, with no need for labeled datasets.
@cite_37 @cite_26 @cite_4 @cite_39 @cite_81 @cite_31 @cite_16
2771471422 2963508807 2337540763 2135533529
Articulated hand pose estimation is a challenging task for human-computer interaction. The state-of-the-art hand pose estimation algorithms work only with one or a few subjects for which they have been calibrated or trained. Particularly, the hybrid methods based on learning followed by model fitting or model based deep learning do not explicitly consider varying hand shapes and sizes. In this work, we introduce a novel hybrid algorithm for estimating the 3D hand pose as well as bone-lengths of the hand skeleton at the same time, from a single depth image. The proposed CNN architecture learns hand pose parameters and scale parameters associated with the bone-lengths simultaneously. Subsequently, a new hybrid forward kinematics layer employs both parameters to estimate 3D joint positions of the hand. For end-to-end training, we combine three public datasets NYU, ICVL and MSRA-2015 in one unified format to achieve large variation in hand shapes and sizes. Among hybrid methods, our method shows improved accuracy over the state-of-the-art on the combined dataset and the ICVL dataset that contain multiple subjects. Also, our algorithm is demonstrated to work well with unseen images. Most of the existing deep learning-based methods for 3D hand and human pose estimation from a single depth map are based on a common framework that takes a 2D depth map and directly regresses the 3D coordinates of keypoints, such as hand or human body joints, via 2D convolutional neural networks (CNNs). The first weakness of this approach is the presence of perspective distortion in the 2D depth map. While the depth map is intrinsically 3D data, many previous methods treat depth maps as 2D images that can distort the shape of the actual object through projection from 3D to 2D space. This compels the network to perform perspective distortion-invariant estimation. The second weakness of the conventional approach is that directly regressing 3D coordinates from a 2D image is a highly nonlinear mapping, which causes difficulty in the learning procedure. To overcome these weaknesses, we firstly cast the 3D hand and human pose estimation problem from a single depth map into a voxel-to-voxel prediction that uses a 3D voxelized grid and estimates the per-voxel likelihood for each keypoint. We design our model as a 3D CNN that provides accurate estimates while running in real-time. Our system outperforms previous methods in almost all publicly available 3D hand and human pose estimation datasets and placed first in the HANDS 2017 frame-based 3D hand pose estimation challenge. The code is available in1. With the increase number of companies focusing on commercializing Augmented Reality (AR), Virtual Reality (VR) and wearable devices, the need for a hand based input mechanism is becoming essential in order to make the experience natural, seamless and immersive. Hand pose estimation has progressed drastically in recent years due to the introduction of commodity depth cameras. Hand pose estimation based on vision is still a challenging problem due to its complexity from self-occlusion (between fingers), close similarity between fingers, dexterity of the hands, speed of the pose and the high dimension of the hand kinematic parameters. Articulated hand pose estimation is still an open problem and under intensive research from both academia and industry. The 2 approaches used for hand pose estimation are: discriminative and generative. Generative approach is a model based that tries to fit a hand model to the observed data. Discriminative approach is appearance based, usually implemented with machine learning (ML) and require a large amount of training data. Recent hand pose estimation uses hybrid approach by combining both discriminative and generative methods into a single hand pipeline. In this paper, we focus on reviewing recent progress of hand pose estimation from depth sensor. We will survey discriminative methods, generative methods and hybrid methods. This paper is not a comprehensive review of all hand pose estimation techniques, it is a subset of some of the recent state-of-the-art techniques. We investigate the task of 2D articulated human pose estimation in unconstrained still images. This is extremely challenging because of variation in pose, anatomy, clothing, and imaging conditions. Current methods use simple models of body part appearance and plausible configurations due to limitations of available training data and constraints on computational expense. We show that such models severely limit accuracy. Building on the successful pictorial structure model (PSM) we propose richer models of both appearance and pose, using state-of-the-art discriminative classifiers without introducing unacceptable computational expense. We introduce a new annotated database of challenging consumer images, an order of magnitude larger than currently available datasets, and demonstrate over 50 relative improvement in pose estimation accuracy over a stateof-the-art method.