Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Volumetric Model <s> This paper describes a framework for constructing a linear subspace model of image appearance for complex articulated 3D figures such as humans and other animals. A commercial motion capture system provides 3D data that is aligned with images of subjects performing various activities. Portions of a limb's image appearance are seen from multiple views and for multiple subjects. From these partial views, weighted principal component analysis is used to construct a linear subspace representation of the "unwrapped" image appearance of each limb. The linear subspaces provide a generative model of the object appearance that is exploited in a Bayesian particle filtering tracking system. Results of tracking single limbs and walking humans are presented. <s> BIB001 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Volumetric Model <s> This paper presents an example-based method for calculating skeleton-driven body deformations. Our example data consists of range scans of a human body in a variety of poses. Using markers captured during range scanning, we construct a kinematic skeleton and identify the pose of each scan. We then construct a mutually consistent parameterization of all the scans using a posable subdivision surface template. The detail deformations are represented as displacements from this surface, and holes are filled smoothly within the displacement maps. Finally, we combine the range scans using k-nearest neighbor interpolation in pose space. We demonstrate results for a human upper body with controllable pose, kinematics, and underlying surface shape. <s> BIB002 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Volumetric Model <s> We describe a method for the acquisition of deformable human geometry from silhouettes. Our technique uses a commercial tracking system to determine the motion of the skeleton, then estimates geometry for each bone using constraints provided by the silhouettes from one or more cameras. These silhouettes do not give a complete characterization of the geometry for a particular point in time, but when the subject moves, many observations of the same local geometries allow the construction of a complete model. Our reconstruction algorithm provides a simple mechanism for solving the problems of view aggregation, occlusion handling, hole filling, noise removal, and deformation modeling. The resulting model is parameterized to synthesize geometry for new poses of the skeleton. We demonstrate this capability by rendering the geometry for motion sequences that were not included in the original datasets. <s> BIB003 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Volumetric Model <s> We introduce the SCAPE method (Shape Completion and Animation for PEople)---a data-driven method for building a human shape model that spans variation in both subject shape and pose. The method is based on a representation that incorporates both articulated and non-rigid deformations. We learn a pose deformation model that derives the non-rigid surface deformation as a function of the pose of the articulated skeleton. We also learn a separate model of variation based on body shape. Our two models can be combined to produce 3D surface models with realistic muscle deformation for different people in different poses, when neither appear in the training set. We show how the model can be used for shape completion --- generating a complete surface mesh given a limited set of markers specifying the target shape. We present applications of shape completion to partial view completion and motion capture animation. In particular, our method is capable of constructing a high-quality animated surface model of a moving person, with realistic muscle deformation, using just a single static scan and a marker motion capture sequence of the person. <s> BIB004 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Volumetric Model <s> During dynamic activities, the surface of the human body moves in many subtle but visually significant ways: bending, bulging, jiggling, and stretching. We present a technique for capturing and animating those motions using a commercial motion capture system and approximately 350 markers. Although the number of markers is significantly larger than that used in conventional motion capture, it is only a sparse representation of the true shape of the body. We supplement this sparse sample with a detailed, actor-specific surface model. The motion of the skin can then be computed by segmenting the markers into the motion of a set of rigid parts and a residual deformation (approximated first as a quadratic transformation and then with radial basis functions). We demonstrate the power of this approach by capturing flexing muscles, high frequency motions, and abrupt decelerations on several actors. We compare these results both to conventional motion capture and skinning and to synchronized video of the actors. <s> BIB005 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Volumetric Model <s> We present a novel algorithm to jointly capture the motion and the dynamic shape of humans from multiple video streams without using optical markers. Instead of relying on kinematic skeletons, as traditional motion capture methods, our approach uses a deformable high-quality mesh of a human as scene representation. It jointly uses an image-based 3D correspondence estimation algorithm and a fast Laplacian mesh deformation scheme to capture both motion and surface deformation of the actor from the input video footage. As opposed to many related methods, our algorithm can track people wearing wide apparel, it can straightforwardly be applied to any type of subject, e.g. animals, and it preserves the connectivity of the mesh over time. We demonstrate the performance of our approach using synthetic and captured real-world video sequences and validate its accuracy by comparison to the ground truth. <s> BIB006 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Volumetric Model <s> Much of the research on video-based human motion capture assumes the body shape is known a priori and is represented coarsely (e.g. using cylinders or superquadrics to model limbs). These body models stand in sharp contrast to the richly detailed 3D body models used by the graphics community. Here we propose a method for recovering such models directly from images. Specifically, we represent the body using a recently proposed triangulated mesh model called SCAPE which employs a low-dimensional, but detailed, parametric model of shape and pose-dependent deformations that is learned from a database of range scans of human bodies. Previous work showed that the parameters of the SCAPE model could be estimated from marker-based motion capture data. Here we go further to estimate the parameters directly from image data. We define a cost function between image observations and a hypothesized mesh and formulate the problem as optimization over the body shape and pose parameters using stochastic search. Our results show that such rich generative models enable the automatic recovery of detailed human shape and pose from images. <s> BIB007 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Volumetric Model <s> Strong lighting is common in natural scenes yet is often viewed as a nuisance for object pose estimation and tracking. In human shape and pose estimation, cast shadows can be confused with foreground structure while self shadowing and shading variation on the body cause the appearance of the person to change with pose. Rather than attempt to minimize the effects of lighting and shadows, we show that strong lighting in a scene actually makes pose and shape estimation more robust. Additionally, by recovering multiple body poses we are able to automatically estimate the lighting in the scene and the albedo of the body. Our approach makes use of a detailed 3D body model, the parameters of which are directly recovered from image data. We provide a thorough exploration of human pose estimation under strong lighting conditions and show: 1. the estimation of the light source from cast shadows; 2. the estimation of the light source and the albedo of the body from multiple body poses; 3. that a point light and cast shadows on the ground plane can be treated as an additional "shadow camera" that improves pose and shape recovery, particularly in monocular scenes. Additionally we introduce the notion of albedo constancy which employs lighting normalized image data for matching. Our experiments with multiple subjects show that rather than causing problems, strong lighting improves human pose and shape estimation. <s> BIB008 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Volumetric Model <s> This paper proposes a method for capturing the performance of a human or an animal from a multi-view video sequence. Given an articulated template model and silhouettes from a multi-view image sequence, our approach recovers not only the movement of the skeleton, but also the possibly non-rigid temporal deformation of the 3D surface. While large scale deformations or fast movements are captured by the skeleton pose and approximate surface skinning, true small scale deformations or non-rigid garment motion are captured by fitting the surface to the silhouette. We further propose a novel optimization scheme for skeleton-based pose estimation that exploits the skeleton's tree structure to split the optimization problem into a local one and a lower dimensional global one. We show on various sequences that our approach can capture the 3D motion of animals and humans accurately even in the case of rapid movements and wide apparel like skirts. <s> BIB009 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Volumetric Model <s> We describe a solution to the challenging problem of estimating human body shape from a single photograph or painting. Our approach computes shape and pose parameters of a 3D human body model directly from monocular image cues and advances the state of the art in several directions. First, given a user-supplied estimate of the subject's height and a few clicked points on the body we estimate an initial 3D articulated body pose and shape. Second, using this initial guess we generate a tri-map of regions inside, outside and on the boundary of the human, which is used to segment the image using graph cuts. Third, we learn a low-dimensional linear model of human shape in which variations due to height are concentrated along a single dimension, enabling height-constrained estimation of body shape. Fourth, we formulate the problem of parametric human shape from shading. We estimate the body pose, shape and reflectance as well as the scene lighting that produces a synthesized body that robustly matches the image evidence. Quantitative experiments demonstrate how smooth shading provides powerful constraints on human shape. We further demonstrate a novel application in which we extract 3D human models from archival photographs and paintings. <s> BIB010 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Volumetric Model <s> We define a new “contour person” model of the human body that has the expressive power of a detailed 3D model and the computational benefits of a simple 2D part-based model. The contour person (CP) model is learned from a 3D SCAPE model of the human body that captures natural shape and pose variations; the projected contours of this model, along with their segmentation into parts forms the training set. The CP model factors deformations of the body into three components: shape variation, viewpoint change and part rotation. This latter model also incorporates a learned non-rigid deformation model. The result is a 2D articulated model that is compact to represent, simple to compute with and more expressive than previous models. We demonstrate the value of such a model in 2D pose estimation and segmentation. Given an initial pose from a standard pictorial-structures method, we refine the pose and shape using an objective function that segments the scene into foreground and background regions. The result is a parametric, human-specific, image segmentation. <s> BIB011 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Volumetric Model <s> We propose a new non-rigid articulated point set registration framework for human pose estimation that aims at improving two recent registration techniques and filling the gap between the two. One is Coherent Point Drift (CPD) that is a powerful Gaussian Mixture Model (GMM) -based non-rigid registration method, but may not be suitable for articulated deformations due to the violation of motion coherence assumption. The other is articulated ICP (AICP) that is effective for human pose estimation but prone to be trapped in local minima without good correspondence initialization. To bridge the gap of the two, a new non-rigid registration method, called Global-Local Topology Preservation (GLTP), is proposed by integrating a Local Linear Embedding (LLE) -based topology constraint with CPD in a GMM-based formulation, which accommodates articulated non-rigid deformations and provides reliable correspondence estimation for AICP initialization. The experiments on both 3D scan data and depth images demonstrate the effectiveness of the proposed framework. <s> BIB012 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Volumetric Model <s> We propose a new 3D model of the human body that is both realistic and part-based. The body is represented by a graphical model in which nodes of the graph correspond to body parts that can independently translate and rotate in 3D and deform to represent different body shapes and to capture pose-dependent shape variations. Pairwise potentials define a “stitching cost” for pulling the limbs apart, giving rise to the stitched puppet (SP) model. Unlike existing realistic 3D body models, the distributed representation facilitates inference by allowing the model to more effectively explore the space of poses, much like existing 2D pictorial structures models. We infer pose and body shape using a form of particle-based max-product belief propagation. This gives SP the realism of recent 3D body models with the computational advantages of part-based models. We apply SP to two challenging problems involving estimating human shape and pose from 3D data. The first is the FAUST mesh alignment challenge, where ours is the first method to successfully align all 3D meshes with no pose prior. The second involves estimating pose and shape from crude visual hull representations of complex body movements. <s> BIB013
Volumetric Models realistically represent 3D body shapes and poses. Geometric shapes and meshes are both effective volumetric models. When using geometric shapes as model components, human body parts are approximated with cylinders, conics, and other shapes, assembling body limbs. For example, a person could be modeled as a composite of cylinders, with each cylinder connected to one or several other cylinders BIB001 . Each joint of the cylinders has 1 to 3 degrees of freedom (DOF). The model is described by the global translation and rotation. The limb pattern is extracted from the model parameters, and the surface space can be determined by solving the least-square problem BIB009 . Conic sections are also utilized to model 3D human limb shapes. Cylindrical and conic sections lead to rectangular or quadrilateral projected shapes. Such models clearly capture the true shape of human limbs given wide variations in anatomy or clothing, and are more accurate than pictorial structure-based approaches. Another way of modeling a volumetric human body is meshes. The meshes are deformable and triangulated models, so they are more suited for the representation of non-rigid human bodies BIB006 . One way to acquire mesh models is through 3D scans BIB002 BIB005 BIB003 . To estimate joint locations, the meshes are usually segmented to several body parts. One widely-used 3D mesh model is Shape Completion and Animation of People (SCAPE) BIB004 BIB010 BIB007  BIB012 . Stitched puppet BIB013 models enhance the SCAPE model by adding pairwise potentials. They define a "stitching cost" for pulling the limbs apart, and learn pairwise relationships from images. Furthermore, 3D human body models are incorporated with shading. For a given mesh, the shape deformation gradients are concatenated into a single column vector. A Blinn-Phong model with diffuse and specular components can be used to approximate a body's reflectance when there is a single light source  . The shadows cast from a point light source provide additional constraints on pose and shape BIB008 . After the pose and shape parameters are estimated, the light position from shadows are determined, and the pose and shape from foreground regionsand shadow regions are also re-estimated. Models that are expressive enough to represent a wide range of human bodies and poses with low dimensions are also explored BIB011 . The authors of BIB004 build on the SCAPE model and develop a factored representation.