Human motion capture and analysis has applications in different fields such as kinesiology, bio-mechanics, surveillance, human-computer interaction, animation and video games. Motion capture for humans describes the activity of analyzing and expressing human motion in mathematical terms. The task of motion capture may be divided into a number of systematically distinct groups, e.g. initialization, tracking, pose estimation, and gesture recognition.
There is a correspondingly large body of developments on human motion analysis and pose estimation from video data. However, the requirements in terms of the detail of pose parameters and accuracy in estimation vary from application to application as does the form of the available input data. Surveillance applications, for instance, usually require just the location of the subject or an approximate estimate of human pose from a single video stream, whereas bio-mechanical applications require detailed pose estimation, including different joint angles, from images obtained using multiple video cameras. Bio-mechanical and clinical applications require accurate capture of normal and pathological human movement without the artifacts associated with current state of the art marker-based motion capture techniques.
The most common methods to accurate capture of 3-D human movement, such as optical, magnetic, and elective mechanical, require attachment of markers, fixtures, or sensors to body segments. These methods are invasive, i.e., they encumber the subject, and require subject preparation time. They may also hinder or influence the movement of the subject due to their invasive nature. Marker-based motion capture systems, although widely used, are not very accurate due to the relative motion between muscle and skin tissue and the underlying skeletal structure. A passive or non-invasive motion capture system composed of video cameras, therefore, possesses several advantages over marker-based systems and is highly desirable.
There exist a number of techniques to estimate the pose of human subjects from both single and multiple cameras. Some of these methods are applied directly to images, while others are applied to 3-D volumetric data (or voxels), which can be computed from image silhouettes. Most methods, however, address applications such as surveillance, human computer interaction, or animation, where accurate joint angle estimation is not necessary or important.
Most approaches use a human body model to guide the pose estimation process, as the use of a model greatly increases the accuracy and robustness of the algorithm. It is therefore necessary to estimate the parameters of the human body model as well. Estimating 3-D pose of an articulated object using images from a single camera is an inherently difficult task due to ill-posed nature of estimating 3-D structure and motion from 2-D images as well as self-occlusion. In order to accurately estimate the 3-D joint angle parameters required in bio-mechanical and clinical applications, it is necessary to use 3-D input data such as voxels in the estimation algorithm. Voxels can be computed from 3-D mesh data obtained from laser scanners as well as images obtained from multiple calibrated cameras.
A large number of pose estimation algorithms use a single image or single image stream to estimate the pose of the subject or use simplified models. Several pose tracking algorithms also assume that the initial pose is known, and use either motion or silhouettes, or alternatively voxels. But few tracking algorithms combine both motion and static cues. The accuracy and the robustness of these algorithms vary as does the suitability of the techniques for different applications. Cedras and Shah in “Motion-based recognition: A survey,” Image and Vision Computing, vol. 13, no. 2, pp. 129-155, March 1995, provide a survey of motion-based recognition methods which require the use of motion data that the markerless motion capture methods can provide. There are several methods (K. Rohr, Human Movement Analysis Based on Explicit Motion Models, Kluwer Academic, 1997; D. Ramanan and D. A. Forsyth “Finding and tracking people from the bottom up.” in CVPR (2), 2003, pp. 467-474; X. Ren, A. C. Berg, and J. Malik, “Recovering human body configurations using pairwise constraints between parts,” in International Conference on Computer Vision, 2005; and G. Mori and J. Malik, “Estimating human body configurations using shape context matching,” in ECCV, 2002, pp. 666-680) to estimate pose from a single view, while I. A. Kakadiaris and D. Metaxas, “3D human body model acquisition from multiple views,” in Fifth International Conference on Computer Vision, 1995, p. 61; I. Mikic, M. Trivedi, E. Hunter, and P. Cosman, “Human body model acquisition and tracking using voxel data,” International Journal of Computer Vision, vol. 53, no. 3, 2003; C. W. Chu, O. C. Jenkins, and M. J. Mataric, “Markerless kinematic model and motion capture from volume sequences: in CVPR (2), 2003, pp. 475-482; K. Cheung, S. Baker, and T. Kanade, “Shape-from-silhouette of articulated objects and its use for human body kinematics estimation and motion capture,” in IEEE CVPR, June 2003; and J. Carranza, C. Theobalt, M. Magnor, and H. Seidel, “Freeviewpoint video of human actors,” ACM Transactions on Graphics, vol. 22, no. 2, pp. 569-577, 2003) estimate the pose from multiple views. Specifically the algorithms presented in I. Mikic, et al., C. W. Chu, et al., K. Cheung, et al., J. Carranza, et al., estimate the pose from voxel representations. Carranza, et al. describe a system that uses multi-view synchronized video footage of an actor's performance to estimate the motion parameters and to interactively re-render the actor's appearance from any viewpoint.
Chu, et al. describes a method for pose estimation using Isomaps (J. B. Tenebaum, V. de Silva, and J. C. Langford, “A global geometric framework for nonlinear dimensionality reduction,” Science, vol. 290, no. 5500, pp. 2319-2323, 2000) to transform the voxels to its pose-invariant intrinsic space representation and obtain a skeleton representation. Cheung, et al. extend shapes-from-silhouette methods to articulated objects. Given silhouettes of a moving articulated object, they propose an iterative algorithm to solve the simultaneous assignment of silhouette points to a body part and alignment of the body part. These methods work well with simple poses, but they are usually unable to handle complex poses where there is self-contact, i.e., one or more of the limbs touches the others.
Anguelov, et al. (“Recovering articulated object models from 3-D range data”, Uncertainty in Artificial Intelligence Conference, 2004) describe an algorithm that automatically decomposes an object into approximately rigid parts, their location, and the underlying articulated structure given a set of meshes describing the object in different poses. They use an unsupervised non-rigid technique to register the meshes and perform segmentation using the EM algorithm. Krahnstoever, et al. (“Articulated models from video,” in Computer vision and Pattern Recognition, 2004, pp. 894-901) address the issue of acquiring articulated models directly from a monocular video. Structure, shape and appearance of articulated models are estimated, but this method is limited in its application as well as in accuracy in extracting complete 3-D human body models as it uses a single camera.
Algorithms that estimate the complete human body model from multiple views are presented in Mikic, et al. and Kakadiaris, et al. For example, the Mikic, et al. propose a model acquisition algorithm using voxels, which starts with a simple body part localization procedure based on fitting and growing template and uses prior knowledge of shapes and dimensions of average body parts. Kakadiaris, et al. present a Human Body Part Identification Strategy (HBPIS) that recovers all the body parts of a moving human based on the spatio-temporal analysis of its deforming silhouette using input from three mutually orthogonal views. However, they specify a protocol of movements that the subject is required to go through.
Brostow, et al., (“Novel skeletal representation for articulated creatures” European Conference on Computer Vision, 2004) present a skeletonization method that uses voxel data to estimate a skeleton representation.
Many methods assume some kind of human body model.
For example, N. I. Badler, et al. (Simulating Humans. Oxford University Press, Oxford, UK, 1993) suggest several methods to represent human subjects in terms of the shape as well as the articulated structure. Modified super-quadrics to represent shapes for the human body model is suggested in D. Gavrila, et al. (“3-D model-based tracking of humans in action: A multi-view approach,” in Computer Vision and Pattern Recognition, 1996, pp. 73-80).
M. Belkin, et al. (“Laplacian eigenmaps for dimensionality reduction and data representation,” Neural Comput., vol. 15, no. 6, pp. 1373-1396, 2003) describe the construction of a representation for data lying in a low dimensional manifold embedded in a high dimensional space and use Laplacian Eigenmaps for dimensionality reduction. While Laplacian eigenmaps and other manifold methods have been applied to dimensionality reduction problems such as classification and Face retrieval using Laplacian faces, none of them, however, map the voxels to a higher dimensional space in order to extract the 1-D structure.
There also exist other methods for dimensionality reduction such as Isomaps, charting a manifold (M. Brand, “Charting a manifold,” in Neural Information Processing Systems, 2002), Kernal Eigenvalue analysis (B. Schölkopf, et al. (“Nonlinear component analysis as a kernel eigenvalue problem,”, Neural Computation, vol. 10, pp. 1299-1319, 1998), the Locally Linear Embedding algorithm (S. T. Rowels, et al. “Nonlinear dimensionality reduction by locally linear embedding,” Science, vol. 290, no. 5500, pp. 2323-2326, 2000) and for reducing articulated objects to pose-invariant structure A. Elad, et al. (“On bending invariant signatures for surface,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 25, pp. 1285-1295, 2003). M. Belkin, et al. analyze the connection of Locally Linear Embedding algorithm proposed by S. T. Rowels, et al. (“Nonlinear dimensionality reduction by locally linear embedding,” Science, vol. 290, no. 5500, pp. 2323-2326, 2000) to the Laplacian.
M. Yamamoto, et al. (“Human motion analysis based on a robot arm model”, CVPR, pages 664-665, 1991) analyze human motion based on a robot model and M. Yamamoto, et al. (“Incremental tracking of human actions from multiple views), CVPR, pages 2-7, 1998) track human motion using multiple cameras. D. M. Gavrila, et al. (“3-D Model-based tracking of human in action,” Computer Vision and Pattern Recognition, pp. 73-80, 1996) present a multi-view approach for 3D model-based tracking of humans in action. They use a generate-and-test algorithm in which they search for poses in a parameter space and match them using a variant of Chamfer matching. R. Plankers, et al. (“Articulated soft objects for video-based body modeling”, ICCV, pages 394-401, 2001) use articulated soft objects with an articulated underlying skeleton as a model, and silhouette data for shape and motion recovery from stereo and trinocular image sequences. Christian Theobalt, et al. (“Combining 3D flow fields with silhouette-based human motion capture for immersive video”, Graph. Models, 66(6):333-351, 2004) project the texture of the model obtained from silhouette-based methods and refine the pose using the flow field. Q. Delamarre, et al. (“3D articulated models and multi-view tracking with silhouettes”, ICCV, pages 716-721, 1999) use 3D articulated models for tracking with silhouettes. They use silhouette contours and apply forces to the contours obtained from the projection of the 3D model so that they move towards the silhouette contours obtained from multiple images. Cheung, et al. (“Shape-from-silhouette of articulated objects and its use for human body kinematics estimation and motion capture”, IEEE Conference on Computer Vision and Pattern Recognition, pages 77-84, June 2003) extend shapes-from-silhouette methods to articulated objects. Given silhouettes of a moving articulated object, they propose an iterative algorithm to solve the simultaneous assignment of silhouette points to a body part and alignment of the body part.
There are also methods that attempt to estimate the pose from a monocular video sequence. They propose different techniques to resolve kinematic ambiguities faced in the monocular pose estimation problem.
Despite extensive research and development in the area of motion capture, further improvements are still needed.
It is highly desirable to have a markerless motion capture technique providing accurate capture of the motion in an effective automated fashion capable of handling poses of any complexity.