Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> A method for the analysis and synthesis of closed curves in the plane is developed using the Fourier descriptors FD's of Cosgriff [1]. A curve is represented parametrically as a function of arc length by the accumulated change in direction of the curve since the starting point. This function is expanded in a Fourier series and the coefficients are arranged in the amplitude/phase-angle form. It is shown that the amplitudes are pure form invariants as well as are certain simple functions of phase angles. Rotational and axial symmetry are related directly to simple properties of the Fourier descriptors. An analysis of shape similarity or symmetry can be based on these relationships; also closed symmetric curves can be synthesized from almost arbitrary Fourier descriptors. It is established that the Fourier series expansion is optimal and unique with respect to obtaining coefficients insensitive to starting point. Several examples are provided to indicate the usefulness of Fourier descriptors as features for shape discrimination and a number of interesting symmetric curves are generated by computer and plotted out. <s> BIB001 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> A method for comparing polygons that is a metric, invariant under translation, rotation, and change of scale, reasonably easy to compute, and intuitive is presented. The method is based on the L/sub 2/ distance between the turning functions of the two polygons. It works for both convex and nonconvex polygons and runs in time O(mn log mn), where m is the number of vertices in one polygon and n is the number of vertices in the other. Some examples showing that the method produces answers that are intuitively reasonable are presented. > <s> BIB002 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> An object recognition system has been developed that uses a new class of local image features. The features are invariant to image scaling, translation, and rotation, and partially invariant to illumination changes and affine or 3D projection. These features share similar properties with neurons in inferior temporal cortex that are used for object recognition in primate vision. Features are efficiently detected through a staged filtering approach that identifies stable points in scale space. Image keys are created that allow for local geometric deformations by representing blurred image gradients in multiple orientation planes and at multiple scales. The keys are used as input to a nearest neighbor indexing method that identifies candidate object matches. Final verification of each match is achieved by finding a low residual least squares solution for the unknown model parameters. Experimental results show that robust object recognition can be achieved in cluttered partially occluded images with a computation time of under 2 seconds. <s> BIB003 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> Example-based methods are effective for parameter estimation problems when the underlying system is simple or the dimensionality of the input is low. For complex and high-dimensional problems such as pose estimation, the number of required examples and the computational complexity rapidly become prohibitively high. We introduce a new algorithm that learns a set of hashing functions that efficiently index examples relevant to a particular estimation task. Our algorithm extends locality-sensitive hashing, a recently developed method to find approximate neighbors in time sublinear in the number of examples. This method depends critically on the choice of hash functions that are optimally relevant to a particular estimation problem. Experiments demonstrate that the resulting algorithm, which we call parameter-sensitive hashing, can rapidly and accurately estimate the articulated pose of human figures from a large database of example images. <s> BIB004 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> We demonstrate that shape contexts can be used to quickly prune a search for similar shapes. We present two algorithms for rapid shape retrieval: representative shape contexts, performing comparisons based on a small number of shape contexts, and shapemes, using vector quantization in the space of shape contexts to obtain prototypical shape pieces. <s> BIB005 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> We study the question of feature sets for robust visual object recognition; adopting linear SVM based human detection as a test case. After reviewing existing edge and gradient based descriptors, we show experimentally that grids of histograms of oriented gradient (HOG) descriptors significantly outperform existing feature sets for human detection. We study the influence of each stage of the computation on performance, concluding that fine-scale gradients, fine orientation binning, relatively coarse spatial binning, and high-quality local contrast normalization in overlapping descriptor blocks are all important for good results. The new approach gives near-perfect separation on the original MIT pedestrian database, so we introduce a more challenging dataset containing over 1800 annotated human images with a large range of pose variations and backgrounds. <s> BIB006 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> Capturing the human hand motion from video involves the estimation of the rigid global hand pose as well as the nonrigid finger articulation. The complexity induced by the high degrees of freedom of the articulated hand challenges many visual tracking techniques. For example, the particle filtering technique is plagued by the demanding requirement of a huge number of particles and the phenomenon of particle degeneracy. This paper presents a novel approach to tracking the articulated hand in video by learning and integrating natural hand motion priors. To cope with the finger articulation, this paper proposes a powerful sequential Monte Carlo tracking algorithm based on importance sampling techniques, where the importance function is based on an initial manifold model of the articulation configuration space learned from motion-captured data. In addition, this paper presents a divide-and-conquer strategy that decouples the hand poses and finger articulations and integrates them in an iterative framework to reduce the complexity of the problem. Our experiments show that this approach is effective and efficient for tracking the articulated hand. This approach can be extended to track other articulated targets. <s> BIB007 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> We introduce a novel set of features for robust object recognition. Each element of this set is a complex feature obtained by combining position- and scale-tolerant edge-detectors over neighboring positions and multiple orientations. Our system's architecture is motivated by a quantitative model of visual cortex. We show that our approach exhibits excellent recognition performance and outperforms several state-of-the-art systems on a variety of image datasets including many different object categories. We also demonstrate that our system is able to learn from very few examples. The performance of the approach constitutes a suggestive plausibility proof for a class of feedforward models of object recognition in cortex. <s> BIB008 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> We present a novel approach that allows us to reliably compute many useful properties of a silhouette. Our approach assigns, for every internal point of the silhouette, a value reflecting the mean time required for a random walk beginning at the point to hit the boundaries. This function can be computed by solving Poisson's equation, with the silhouette contours providing boundary conditions. We show how this function can be used to reliably extract various shape properties including part structure and rough skeleton, local orientation and aspect ratio of different parts, and convex and concave sections of the boundaries. In addition to this, we discuss properties of the solution and show how to efficiently compute this solution using multigrid algorithms. We demonstrate the utility of the extracted properties by using them for shape classification and retrieval. <s> BIB009 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> We integrate the cascade-of-rejectors approach with the Histograms of Oriented Gradients (HoG) features to achieve a fast and accurate human detection system. The features used in our system are HoGs of variable-size blocks that capture salient features of humans automatically. Using AdaBoost for feature selection, we identify the appropriate set of blocks, from a large set of possible blocks. In our system, we use the integral image representation and a rejection cascade which significantly speed up the computation. For a 320 × 280 image, the system can process 5 to 30 frames per second depending on the density in which we scan the image, while maintaining an accuracy level similar to existing methods. <s> BIB010 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> Recovering the pose of a person from single images is a challenging problem. This paper discusses a bottom-up approach that uses local image features to estimate human upper body pose from single images in cluttered backgrounds. The method takes the image window with a dense grid of local gradient orientation histograms, followed by non negative matrix factorization to learn a set of bases that correspond to local features on the human body, enabling selective encoding of human-like features in the presence of background clutter. Pose is then recovered by direct regression. This approach allows us to key on gradient patterns such as shoulder contours and bent elbows that are characteristic of humans and carry important pose information, unlike current regressive methods that either use weak limb detectors or require prior segmentation to work. The system is trained on a database of images with labelled poses. We show that it estimates pose with similar performance levels to current example-based methods, but unlike them it works in the presence of natural backgrounds, without any prior segmentation. <s> BIB011 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> Histograms of local appearance descriptors are a popular representation for visual recognition. They are highly discriminant and have good resistance to local occlusions and to geometric and photometric variations, but they are not able to exploit spatial co-occurrence statistics at scales larger than their local input patches. We present a new multilevel visual representation, 'hyperfeatures', that is designed to remedy this. The starting point is the familiar notion that to detect object parts, in practice it often suffices to detect co-occurrences of more local object fragments - a process that can be formalized as comparison (e.g. vector quantization) of image patches against a codebook of known fragments, followed by local aggregation of the resulting codebook membership vectors to detect co-occurrences. This process converts local collections of image descriptor vectors into somewhat less local histogram vectors - higher-level but spatially coarser descriptors. We observe that as the output is again a local descriptor vector, the process can be iterated, and that doing so captures and codes ever larger assemblies of object parts and increasingly abstract or 'semantic' image properties. We formulate the hyperfeatures model and study its performance under several different image coding methods including clustering based Vector Quantization, Gaussian Mixtures, and combinations of these with Latent Dirichlet Allocation. We find that the resulting high-level features provide improved performance in several object image and texture image classification tasks. <s> BIB012 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> In this paper we introduce a 3-dimensional (3D) SIFT descriptor for video or 3D imagery such as MRI data. We also show how this new descriptor is able to better represent the 3D nature of video data in the application of action recognition. This paper will show how 3D SIFT is able to outperform previously used description methods in an elegant and efficient manner. We use a bag of words approach to represent videos, and present a method to discover relationships between spatio-temporal words in order to better describe the video data. <s> BIB013 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> In this paper, we address the problem of detecting pedestrians in still images. We introduce an algorithm for learning shapelet features, a set of mid-level features. These features are focused on local regions of the image and are built from low-level gradient information that discriminates between pedestrian and non-pedestrian classes. Using Ad-aBoost, these shapelet features are created as a combination of oriented gradient responses. To train the final classifier, we use AdaBoost for a second time to select a subset of our learned shapelets. By first focusing locally on smaller feature sets, our algorithm attempts to harvest more useful information than by examining all the low-level features together. We present quantitative results demonstrating the effectiveness of our algorithm. In particular, we obtain an error rate 14 percentage points lower (at 10-6 FPPW) than the previous state of the art detector of Dalal and Triggs on the INRIA dataset. <s> BIB014 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> Recent research in visual inference from monocular images has shown that discriminatively trained image-based predictors can provide fast, automatic qualitative 3D reconstructions of human body pose or scene structure in real-world environments. However, the stability of existing image representations tends to be perturbed by deformations and misalignments in the training set, which, in turn, degrade the quality of learning and generalization. In this paper we advocate the semi-supervised learning of hierarchical image descriptions in order to better tolerate variability at multiple levels of detail. We combine multilevel encodings with improved stability to geometric transformations, with metric learning and semi-supervised manifold regularization methods in order to further profile them for task-invariance -resistance to background clutter and within the same human pose class differences. We quantitatively analyze the effectiveness of both descriptors and learning methods and show that each one can contribute, sometimes substantially, to more reliable 3D human pose estimates in cluttered images. <s> BIB015 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> A novel consistent max-covering method is proposed for human pose estimation. We focus on problems in which a rough foreground estimation is available. Pose estimation is formulated as a jigsaw puzzle problem in which the body part tiles maximally cover the foreground region, match local image features, and satisfy body plan and color constraints. This method explicitly imposes a global shape constraint on the body part assembly. It anchors multiple body parts simultaneously and introduces hyperedges in the part relation graph, which is essential for detecting complex poses. Using multiple cues in pose estimation, our method is resistant to cluttered foregrounds. We propose an efficient linear method to solve the consistent max-covering problem. A two-stage relaxation finds the solution in polynomial time. Our experiments on a variety of images and videos show that the proposed method is more robust than previous locally constrained methods. <s> BIB016 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> Some articulated motion representations rely on frame-wise abstractions of the statistical distribution of low-level features such as orientation, color, or relational distributions. As configuration among parts changes with articulated motion, the distribution changes, tracing a trajectory in the latent space of distributions, which we call the configuration space. These trajectories can then be used for recognition using standard techniques such as dynamic time warping. The core theory in this paper concerns embedding the frame-wise distributions, which can be looked upon as probability functions, into a low-dimensional space so that we can estimate various meaningful probabilistic distances such as the Chernoff, Bhattacharya, Matusita, Kullback-Leibler (KL) or symmetric-KL distances based on dot products between points in this space. Apart from computational advantages, this representation also affords speed-normalized matching of motion signatures. Speed normalized representations can be formed by interpolating the configuration trajectories along their arc lengths, without using any knowledge of the temporal scale variations between the sequences. We experiment with five different probabilistic distance measures and show the usefulness of the representation in three different contexts - sign recognition (with large number of possible classes), gesture recognition (with person variations), and classification of human-human interaction sequences (with segmentation problems). We find the importance of using the right distance measure for each situation. The low-dimensional embedding makes matching two to three times faster, while achieving recognition accuracies that are close to those obtained without using a low-dimensional embedding. We also empirically establish the robustness of the representation with respect to low-level parameters, embedding parameters, and temporal-scale parameters. <s> BIB017 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> We present a novel approach to measuring similarity between shapes and exploit it for object recognition. In our framework, the measurement of similarity is preceded by: (1) solving for correspondences between points on the two shapes; (2) using the correspondences to estimate an aligning transform. In order to solve the correspondence problem, we attach a descriptor, the shape context, to each point. The shape context at a reference point captures the distribution of the remaining points relative to it, thus offering a globally discriminative characterization. Corresponding points on two similar shapes will have similar shape contexts, enabling us to solve for correspondences as an optimal assignment problem. Given the point correspondences, we estimate the transformation that best aligns the two shapes; regularized thin-plate splines provide a flexible class of transformation maps for this purpose. The dissimilarity between the two shapes is computed as a sum of matching errors between corresponding points, together with a term measuring the magnitude of the aligning transform. We treat recognition in a nearest-neighbor classification framework as the problem of finding the stored prototype shape that is maximally similar to that in the image. Results are presented for silhouettes, trademarks, handwritten digits, and the COIL data set. <s> BIB018 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> The Scale-Invariant Feature Transform (or SIFT) algorithm is a highly robust method to extract and consequently match distinctive invariant features from images. These features can then be used to reliably match objects in diering images. The algorithm was rst proposed by Lowe [12] and further developed to increase performance resulting in the classic paper [13] that served as foundation for SIFT which has played an important role in robotic and machine vision in the past decade. <s> BIB019 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> We present an approach for automatic 3D human pose reconstruction from monocular images, based on a discriminative formulation with latent segmentation inputs. We advanced the field of structured prediction and human pose reconstruction on several fronts. First, by working with a pool of figure-ground segment hypotheses, the prediction problem is formulated in terms of combined learning and inference over segment hypotheses and 3D human articular configurations. Beside constructing tractable formulations for the combined segment selection and pose estimation problem, we propose new augmented kernels that can better encode complex dependencies between output variables. Furthermore, we provide primal linear re-formulations based on Fourier kernel approximations, in order to scale-up the non-linear latent structured prediction methodology. The proposed models are shown to be competitive in the HumanEva benchmark and are also illustrated in a clip collected from a Hollywood movie, where the model can infer human poses from monocular images captured in complex environments. <s> BIB020 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> We present an algorithm for 3d pose estimation of articulated people in natural images. The poses are disassembled into a collection of local patches and a new pose is inferred by assembling the local patches. This concept allows inference of a wide variety of poses from a small number of training patches. The actual process is realized efficiently by a novel voting scheme where each local patch extracted from the input image is matched to the model patches and matchings cast votes for possible locations and poses of the human body, yielding a set of candidate location-pose pairs. Each candidate is then holistically verified using a top-down model based method, where SVM regression computes the final score by aggregating several scores capturing different features of the candidates. We evaluate our method on both real and synthetic images and demonstrate its ability. <s> BIB021 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> Automatic estimation of 3D shape similarity from video is a very important factor for human action analysis, but also a challenging task due to variations in body topology and the high dimensionality of the pose configuration space. We consider the problem of 3D shape similarity in 3D video sequence for different actors and motions. Most current approaches use conventional global features as a shape descriptor and define the shape similarity using L2 distance. However, such methods are limited to coarse representation and do not sufficiently reflect the pose similarity of human perception. In this paper, we present a novel 3D human pose descriptor called Extremal Human Curves (EHC), extracted from both the spatial and the topological dimensions of body surface. To compare tow shapes, we use an elastic metric in Shape Space between their descriptors, based on static features, and then perform temporal convolutions, thereby capturing the pose information encoded in multiple adjacent frames. We quantitatively analyze the effectiveness of our descriptors for both 3D shape similarity in video and content-based pose retrieval for static shape, and show that each one can contribute, sometimes substantially, to more reliable human shape and pose analysis. Experimental results are promising and show the robustness and accuracy of the proposed approach by comparing the recognition performance against several state-of-the-art methods. <s> BIB022 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> Recent studies have demonstrated that high-level semantics in data can be captured using sparse representation. In this paper, we propose an approach to human body pose estimation in static images based on sparse representation. Given a visual input, the objective is to estimate 3D human body pose using feature space information and geometrical information of the pose space. On the assumption that each data point and its neighbors are likely to reside on a locally linear patch of the underlying manifold, our method learns the sparse representation of the new input using both feature and pose space information and then estimates the corresponding 3D pose by a linear combination of the bases of the pose dictionary. Two strategies for dictionary construction are presented: (i) constructing the dictionary by randomly selecting the frames of a sequence and (ii) selecting specific frames of a sequence as dictionary atoms. We analyzed the effect of each strategy on the accuracy of pose estimation. Extensive experiments on datasets of various human activities show that our proposed method outperforms state-of-the-art methods. <s> BIB023 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> In this paper, we address the problem of 3D human body pose estimation from depth images acquired by a stereo camera. Compared to the Kinect sensor, stereo cameras work outdoors having a much higher operational range, but produce noisier data. In order to deal with such data, we propose a framework for 3D human pose estimation that relies on random forests. The first contribution is a novel grid-based shape descriptor robust to noisy stereo data that can be used by any classifier. The second contribution is a two step classification procedure, first classifying the body orientation, then proceeding with determining the full 3D pose within this orientation cluster. To validate our method, we introduce a dataset recorded with a stereo camera synchronized with an optical motion capture system that provides ground truth human body poses. <s> BIB024 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> In this paper, we propose a deep convolutional neural network for 3D human pose estimation from monocular images. We train the network using two strategies: (1) a multi-task framework that jointly trains pose regression and body part detectors; (2) a pre-training strategy where the pose regressor is initialized using a network trained for body part detection. We compare our network on a large data set and achieve significant improvement over baseline methods. Human pose estimation is a structured prediction problem, i.e., the locations of each body part are highly correlated. Although we do not add constraints about the correlations between body parts to the network, we empirically show that the network has disentangled the dependencies among different body parts, and learned their correlations. <s> BIB025 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> We propose an heterogeneous multi-task learning framework for human pose estimation from monocular image with deep convolutional neural network. In particular, we simultaneously learn a pose-joint regressor and a sliding-window body-part detector in a deep network architecture. We show that including the body-part detection task helps to regularize the network, directing it to converge to a good solution. We report competitive and state-of-art results on several data sets. We also empirically show that the learned neurons in the middle layer of our network are tuned to localized body parts. <s> BIB026 </s> Human Pose Estimation from Monocular Images: A Comprehensive Survey <s> Mid-Level Features <s> The objective of this work is human pose estimation in videos, where multiple frames are available. We investigate a ConvNet architecture that is able to benefit from temporal context by combining information across the multiple frames using optical flow. To this end we propose a network architecture with the following novelties: (i) a deeper network than previously investigated for regressing heatmaps, (ii) spatial fusion layers that learn an implicit spatial model, (iii) optical flow is used to align heatmap predictions from neighbouring frames, and (iv) a final parametric pooling layer which learns to combine the aligned heatmaps into a pooled confidence map. We show that this architecture outperforms a number of others, including one that uses optical flow solely at the input layers, one that regresses joint coordinates directly, and one that predicts heatmaps without spatial fusion. The new architecture outperforms the state of the art by a large margin on three video pose estimation datasets, including the very challenging Poses in the Wild dataset, and outperforms other deep methods that don't use a graphical model on the single-image FLIC benchmark (and also [5, 35] in the high precision region). <s> BIB027
Extracted silhouette features are usually encoded as Fourier descriptors BIB001 , shape contexts BIB005 , geometric signatures BIB002 , Poisson features BIB009 , and so on. The most frequently used shape context descriptor captures the distribution of points relative to the current point being described, as shown in Figure 5a . Specifically, a histogram is computed using log-polar coordinates, and the space is divided into several angle and radius bins. Points falling in each bin are accumulated to form a histogram distribution, as shown in Figure 5b . It converts distributed points into a multi-dimensional descriptor, and this statistical means of computation is robust against local silhouette segmentation errors BIB018 BIB005  BIB016 BIB023 . Other features based on edges or gradients are encoded as histograms of oriented gradients (HOG) BIB006 BIB010 BIB004 , relational edge distribution BIB017 , Scale Invariant Feature Transform (SIFT) BIB003 BIB019 and SIFT-like features BIB011 BIB013 , edgelet features BIB007 , shapelet features BIB014 , and so on. By measuring on a number of scales, SIFT features (shown in Figure 6a ) can be matched against scale variance and are extremely popular among computer vision researchers before deep convolution networks are widely applied to automatically extract features. HOG features are extremely popular features for human pose estimation, and usually several HOG templates representing various states of a body part are learned (visualized in Figure 6b ). Edgelet (in Figure 7 ) and shapelet (in Figure 8 ) features are combinations of edges and gradients, respectively. Other than local features mentioned above, there are many global features which capture overall characteristics, for example, the object foreground map BIB016 and dense grid features, like the grids of HOG descriptors BIB006 or grids of SIFT features BIB011 BIB020 . Grid features-for example, grid of SIFT-outperform the SIFT feature extractor and descriptor, according to experience. Multilevel hierarchical encodings, like Hierarchical Model and X (HMAX) BIB008 , hyperfeatures BIB012 , spatial pyramid  , vocabulary tree, and Multilevel Spatial Blocks (MSB) BIB015 are more stable in preserving invariance to geometric transformations. Other features, such as local paths BIB021 , prediction pipeline BIB024 , and Extremal Human Curves BIB022 are also common features in human pose estimation. A convolutional neural network (CNN, or ConvNet) is currently the most popular feature in computer vision, artificial intelligence, machine learning, and many other fields. CNN is an extension of a neural network. Input images are processed by convolution and downsampled several times to extract features, and fully-connected layers consider integrated efforts from all. Estimated errors are back-propagated, and network parameters are adjusted accordingly. Recently, many works have used CNN extracted features for human pose estimation BIB025 BIB026 BIB027 .