Articulated Pose Estimation by a Graphical Model with Image Dependent Pairwise Relations

We present a method for estimating articulated human pose from a single static image based on a graphical model with novel pairwise relations that make adaptive use of local image measurements. More precisely, we specify a graphical model for human pose which exploits the fact the local image measurements can be used both to detect parts (or joints) and also to predict the spatial relationships between them (Image Dependent Pairwise Relations). These spatial relationships are represented by a mixture model. We use Deep Convolutional Neural Networks (DCNNs) to learn conditional probabilities for the presence of parts and their spatial relationships within image patches. Hence our model combines the representational flexibility of graphical models with the efficiency and statistical power of DCNNs. Our method significantly outperforms the state of the art methods on the LSP and FLIC datasets and also performs very well on the Buffy dataset without any training.


Introduction
Articulated pose estimation is one of the fundamental challenges in computer vision. Progress in this area can immediately be applied to important vision tasks such as human tracking [1], action recognition [19] and video analysis.
Most work on pose estimation has been based on graphical model [7,5,22,21]. The graph nodes represent the body parts (or joints), and the edges model the pairwise relationships between the parts. The score function, or energy, of the model contains unary terms at each node which capture the local appearance cues of the part, and pairwise terms defined at the edges which capture the local contextual relations between the parts. Recently, DeepPose [17] advocates modeling pose in a holistic manner and captures the full context of all body parts in a Deep Convolutional Neural Network (DCNN) [10] based regressor.
In this paper, we present a graphical model with image dependent pairwise relations (IDPRs). As illustrated in Figure 1, we can reliably predict the relative positions of a part's neighbors (as well as the presence of the part itself) by only observing the local image patch around it. So in our model the local image patches give input to both the unary and pairwise terms. This gives stronger pairwise terms because data independent relations are typically either too loose to be helpful or too strict to model highly variable poses.
Our approach requires us to have a method that can extract information about pairwise part relations, as well as part presence, from local image patches. We require this method to be efficient and to share features between different parts and part relationships. To do this, we train a DCNN to output estimates for the part presence and spatial relationships which are used in our unary and pairwise terms of our score function. The weight parameters of different terms in the model are trained using Structured Supported Vector Machine (S-SVM) [18]. In summary, our model combines the representational flexibility of graphical models, including the ability to represent spatial relationships, with the data driven power of DCNNs.
We perform experiments on two standard pose estimation benchmarks: LSP dataset [9] and FLIC dataset [14]. Our method outperforms the state of the art methods by a significant margin on both datasets. We also do cross-dataset evaluation on Buffy dataset [6] (without training on this dataset) and obtain strong results which shows the ability of our model to generalize.

The Model
The Graphical Model and its Variables: We represent human pose by a graphical model G = (V, E) where the nodes V specify the positions of the parts (or joints) and the edges E indicates which parts are spatially related. For simplicity, we impose that the graph structure forms a K−node tree, where K = |V|. The positions of the parts are denoted by l, where l i = (x, y) specifies the pixel location of part i, for i ∈ {1, . . . , K}. For each edge in the graph (i, j) ∈ E, we specify a discrete set of spatial relationships indexed by t ij , which corresponds to a mixture of different spatial relationships (see Figure 1). We denote the set of spatial relationships by t = {t ij , t ji |(i, j) ∈ E}. The image is written as I. We will define a score function F (l, t|t) as follows as a sum of unary and pairwise terms.
Unary Terms: The unary terms give local evidence for part i ∈ V to lie at location l i and is based on the local image patch I(l i ). They are of form: where φ(.|.; θ) is the (scalar-valued) appearance term with θ as its parameters (specified in the next section), and w i is a scalar weight parameter.
Image Dependent Pairwise Relational (IDPR) Terms: These IDPR terms capture our intuition that neighboring parts (i, j) ∈ E can roughly predict their relative spatial positions using only local information (see Figure 1). In our model, the relative positions of parts i and j are discretized into several types t ij ∈ {1, . . . , T ij } (i.e., a mixture of different relationships) with corresponding mean relative positions r tij ij plus small deformations which are modeled by the standard quadratic deformation term. More formally, the pairwise relational score of each edge (i, j) ∈ E is given by: where ψ(∆l = [∆x, ∆y]) = [∆x ∆x 2 ∆y ∆y 2 ] are the standard quadratic deformation features, ϕ(.|.; θ) is the Image Dependent Pairwise Relational (IDPR) term with θ as its parameters (specified in the next section), and w tij ij , w ij , w tji ji , w ji are the weight parameters. The notation ., . specifies dot product and boldface indicates a vector.
The Full Score: The full score F (l, t|I) is a function of the part locations l, the pairwise relation types t, and the input image I. It is expressed as the sum of the unary and pairwise terms: where w 0 is a scalar weight on constant 1 (i.e., the bias term).
The model consists of three sets of parameters: the mean relative positions r = {r tij ij , r tji ji |(i, j) ∈ E} of different pairwise relation types; the parameters θ of the appearance terms and IDPR terms; and the weight parameters w (i.e., w i , w tij ij , w ij , w tji ji , w ji and w 0 ). See Section 4 for the learning of these parameters.

Image Dependent Terms and DCNNs
The appearance terms and IDPR terms depend on the image patches. In other words, a local image patch I(l i ) not only gives evidence for the presence of a part i, but also about the relationship t ij between it and its neighbors j ∈ N(i), where j ∈ N(i) if, and only if, (i, j) ∈ E. This requires us to learn distribution for the state variables i, t ij conditioned on the image patches I(l i ). In order to specify this distribution we must define the state space more precisely, because the number of pairwise spatial relationships varies for different parts with different numbers of neighbors (see Figure 1), and we need also consider the possibility that the patch does not contain a part.
We define c to be the random variable which denotes which part is present c = i for i ∈ {1, ..., K} or c = 0 if no part is present (i.e., the background). We define m cN(c) to be the random variable that determines the spatial relation types of c and takes values in M cN(c) . If c = i has one neighbor j (e.g., the wrist), then M iN(i) = {1, . . . , T ij }. If c = i has two neighbors j and k (e.g., the elbow), The full space is represented as: The size of the space is |S| = K c=0 |M cN(c) |. Each element in this space corresponds to a part with all the types of its pairwise relationships, or the background.
We use DCNN [10] to learn the conditional probability distribution p(c, m cN(c) |I(l i ); θ). DCNN is suitable for this task because it is very efficient and enables us to share features. See section 4 for more details.

Relationship to other models
We now briefly discuss how our method relates to standard models.
Pictorial Structure: We recover pictorial structure models [5] by only allowing one relationship type (i.e., T ij = 1). In this case, our IDPR term conveys no information. Our model reduces to standard unary and (image independent) pairwise terms. The only slight difference is that we use DCNN to learn the unary terms instead of using HOG filters.
Mixtures-of-parts: [21] describes a model with a mixture of templates for each part, where each template is called a "type" of the part. The "type" of each part is defined by its relative position with respect to its parent. This can be obtained by restricting each part in our model to only predict the relative position of its parent (i.e., T ij = 1, if j is not parent of i). In this case, each part is associated with only one informative IDPR term, which can be merged with the appearance term of each part to define different "types" of part in [21]. Also this method does not use DCNNs.

Inference
To detect the optimal configuration for each person, we search for the configurations of the locations l and types t that maximize the score function: (l * , t * ) = arg max l,t F (l, t|I). Since our relational graph is a tree, this can be done efficiently via dynamic programming.
Let K(i) be the set of children of part i in the graph (K(i) = ∅, if part i is a leaf), and S i (l i |I) be maximum score of the subtree rooted at part i with part i located at l i . The maximum score of each subtree can be computed as follow: Using Equation 7, we can recursively compute the overall best score of the model, and the optimal configuration of locations and types can be recovered by the standard backward pass of dynamic programming.
Computation: Since our pairwise term is a quadratic function of locations, l i and l j , the max operation over l k in Equation 7 can be accelerated by using the generalized distance transforms [5]. The resulting approach is very efficient, taking O(T 2 LK) time once the image dependent terms are computed, where T is the number of relation types, L is the total number of locations, and K is the total number of parts in the model. This analysis assumes that all the pairwise spatial relationships have the same number of types, i.e., The computation of the image dependent terms is also efficient. They are computed over all the locations by a single DCNN. Applying DCNN in a sliding fashion is inherently efficient, since the computations common to overlapping regions are naturally shared [16].

Learning
Now we consider the problem of learning the model parameters from images with labeled part locations, which is the data available in most of the human pose datasets [13,6,9,14]. We derive type labels t ij from part location annotations and adopt a supervised approach to learn the model.
Our model consists of three sets of parameters: the mean relative positions r of different pairwise relation types; the parameters θ of the image dependent terms; and the weight parameters w. They are learnt separately by the K-means algorithm for r, DCNN for θ, and S-SVM for w.
Mean Relative Positions and Type Labels: Given the labeled positive images {(I n , l n )} N n=1 , let d ij be the relative position from part i to its neighbor j. We cluster the relative positions over the training set {d n ij } N n=1 to get T ij clusters (in the experiments T ij = 11 for all pairwise relations). Each cluster corresponds to a set of instances of part i that share similar spatial relationship with its neighbor part j. Thus we define each cluster as a pairwise relation type t ij from part i to j in our model, and use the center of each cluster as the mean relative position r tij ij associated with each type. In this way, the mean relative positions of different pairwise relation types are learnt, and the type label t n ij for each training instance is derived based on its cluster index. We use K-means in our experiments by setting K = T ij to do the clustering.
Parameters of Image Dependent Terms: After deriving type labels, each local image patch I(l n ) centered at an annotated part location is labeled with category label c n ∈ {1, . . . , K}, that indicates which part is present, and also the type labels m n c n N(c n ) that indicate its relation types with all its neighbors. In this way, we get a set of labelled patches {I(l n ), c n , m n c n N(c n ) } KN n=1 from positive images (each positive image provides K part patches), and also a set of background patches {I(l n ), 0, 0} sampled from negative images.
Given the labelled part patches and background patches, we train a multi-class DCNN classifier by standard stochastic gradient descent using softmax loss. The DCNN consists of five convolutional layers, 2 max-pooling layers and three fully-connected layers with a final |S| dimensions softmax output, which is defined as our conditional probability distribution, i.e., p(c, m cN(c) |I(l i ); θ). The architecture of our network is summarized in Figure 2.
Weight Parameters: Each pose in the positive image is now labeled with annotated part locations and derived type labels: (I n , l n , t n ). We use S-SVM to learn the weight parameters w. The structure prediction problem is simplified by using 0 − 1 loss, that is all the training examples either have all dimensions of its labels correct or all dimensions of its labels wrong. We denote the former ones as pos examples, and the later ones as neg examples. Since the full score function (Equation 3) is linear in the weight parameters w, we write the optimization function as: where y n ∈ {1, −1}, and Φ(I n , l n , t n ) is a sparse feature vector representing the n-th example and is the concatenation of the image dependent terms (calculated from the learnt DCNN), spatial deformation features, and constant 1. Here y n = 1 if n ∈ pos, and y n = −1 if n ∈ neg.

Experiment
This section introduces the datasets, clarifies the evaluation metrics, describes our experimental setup, presents comparative evaluation results and gives diagnostic experiments.

Datasets and Evaluation Metrics
We perform our experiments on two publicly available human pose estimation benchmarks: (i) the "Leeds Sports Poses" (LSP) dataset [9], that contains 1000 training and 1000 testing images from sport activities with annotated full-body human poses; (ii) the "Frames Labeled In Cinema" (FLIC) dataset [14] that contains 3987 training and 1016 testing images from Hollywood movies with annotated upper-body human poses. We follow previous work and use the observer-centric annotations on both benchmarks. To train our models, we also use the negative training images from the INRIAPerson dataset [2] (These images do not contain people).
We use the most popular evaluation metrics to allow comparison with previous work. Percentage of Correct Parts (PCP) is the standard evaluation metric on several benchmarks including the LSP dataset. However, as discussed in [21], there are several alternative interpretations of PCP that can lead to severely different results. In our paper, we use the stricter version unless otherwise stated, that is we evaluate only a single highest-scoring estimation result for one test image and a body part is considered as correct if both of its segment endpoints (joints) lie within 50% of the length of the ground-truth annotated endpoints (Each test image on the LSP dataset contains only one annotated person). We refer to this version of PCP as strict PCP.
On the FLIC dataset, we use the evaluation metric specified with it [14]: Percentage of Detected Joints (PDJ). PDJ measures the performance using a curve of the percentage of correctly localized joints by varying localization precision threshold. The localization precision threshold is normalized by the scale (defined as distance between left shoulder and right hip) of each ground truth pose to make it scale invariant. We also include the Area Under Curve (AUC) number as the overall evaluation metric of the curve. There are multiple people in the FLIC images, so each ground truth person is also annotated with a torso detection box. During evaluation, we return a single highestscoring estimation result for each ground truth person by restricting our neck part to be localized inside a window defined by the provided torso box.  Figure 2: Architectures of our DCNNs. The size of input patch is 36 × 36 pixels on the LSP dataset, and 72 × 72 pixels on the FLIC dataset. The DCNNs consist of five convolutional layers, 2 max-pooling layers and three fully-connected (dense) layers with a final |S| dimensions output. We use dropout, local response normalization (norm) and overlapping pooling (pool) described in [10].

Implementation detail
Data Augmentation: Our DCNN has millions of parameters, while only several thousand of training images are available. In order to reduce overfitting, we augment the training data by rotating the positive training images through 360 • by steps of size of 10 • . This increases the number of training images by a factor of 36. Although the rotated poses may not be realistic as a whole, each individual part still realistically interacts with its neighbors. For example, a running person rotated by 180 • may seem strange, but the local patches around elbows, wrists, hips etc. are still fairly suitable for telling which parts are present and where the neighboring parts should be. These images are also horizontally flipped to double the training images. We hold out 1000 random pairs of positive images and their flipped version as a validation set for the DCNN training. Also the weight parameters w are trained on this held out set to reduce overfitting to training data.
Note that our DCNN is trained using local part patches and background patches instead of the whole images. This naturally increases the number of training examples by a factor of K (the number of parts). Although the number of dimensions of the DCNN final output also increases linearly with the number of parts, the number of parameters only slightly increase in the last fully-connected layer. This is because most of the parameters are shared between different parts, and thus we can benefit from larger K by having more training examples per parameter. In our experiments, we increase K by adding the midway points between annotated parts, which results in 26 parts on the LSP dataset and 18 parts on the FLIC dataset. Covering a person by more parts also reduces the distance between neighboring parts, which is good for modeling foreshortening [21].
Graph Structure: We define a full-body graph structure for the LSP dataset, and a upper-body graph structure for the FLIC dataset respectively. The graph connects the annotated parts and their midways points to form a tree (See the skeletons in Figure 6 for clarification).
Settings: We use the same number of types for all pairs of neighbors for simplicity. We set it as 11 on all datasets (i.e., T ij = T ji = 11, ∀(i, j) ∈ E), which results in 11 spatial relation types for the parts with one neighbor (e.g., the wrist), 11 2 spatial relation types for the parts with two neighbors (e.g., the elbow), and so forth (recall Figure 1). The patch size of each part is set as 36 × 36 pixels on the LSP dataset, and 72×72 pixels on the FLIC dataset, as the FLIC images are of higher resolution. We use similar DCNN architectures on both datasets, which only differ in the first layer because of different input patch sizes. Figure 2 visualizes the architectures we used. We use the Caffe [8] implementation of DCNN in our experiments.

Benchmark results
We show strict PCP results on the LSP dataset in Table 1, and PDJ results on the FLIC dataset in Figure 3. We also show the PDJ results on the LSP for arms and legs, which are the most challenging parts, in Figure 4. As is shown, our method outperforms state of the art methods by a significant margin on both datasets (see the captions for detailed analysis). Figure 6 shows Figure 3: Comparison of PDJ curves of elbows and wrists on the FLIC dataset. We also show the Area Under Curve (AUC) numbers. Our method significantly outperforms DeepPose [17] and MODEC [14].

Diagnostic Experiments
We perform diagnostic experiments to show the cross-dataset generalization ability of our model, and better understand the influence of each term in our model.

Cross-dataset Generalization:
We directly apply the trained model on the FLIC dataset to the official test set of Buffy dataset [6] (i.e., no training on the Buffy dataset), which also contains upper-body human poses. The Buffy test set includes a subset of people whose upper-body can be detected. We get the newest detection windows from [4], and compare our results to previously published work on this subset.
Most previous work was evaluated with the official evaluation toolkit of Buffy, which uses a less strict PCP implementation, where a part is considered correctly localized if the average distance between its endpoints (joints) and ground-truth is less than 50% of the length of the ground-truth annotated endpoints. We refer to this version of PCP as Buffy PCP and report it along with the strict PCP in Table 2. We also show the PDJ curves in Figure 5. As is shown by both criterions, our cross-dataset performance significantly outperforms the state of the art methods, which shows the good generalization ability of our method. Also note that both DeepPose [17] and our method are trained on the FLIC dataset. Compared with Figure 3, the margin between our method and DeepPose significantly increases in Figure 5, which implies that our method generalizes better to the Buffy dataset.   Terms Analysis: We design two experiments to better understand the influence of each term in our model on the performance. In the first experiment, we use only the unary terms and thus all the parts are localized independently. In the second experiment, we replace the IDPR terms with image independent priors (i.e., in Equation 2, w ij ϕ(t ij |I(l i ); θ) and w ji ϕ(t ji |I(l j ); θ) are replaced with scalar prior terms b tij ij and b tji ji respectively), and retrain the weight parameters along with the new prior terms. In this case, our pairwise relational terms do not depend on the image, but instead is a mixture of Gaussian deformations with image independent biases. We refer to the first experiment as Unary-Only and the second one as No-IDPRs, short for No IDPR terms. The experiments are done on the LSP dataset using identical appearance terms for fair comparison. We show strict PCP results in Table 3. As is shown, all terms in our model significantly improve the performance (see the caption for more analysis). Figure 6: Results on the LSP and FLIC datasets. We show the part localization results along with the graph skeleton we used in the model. The first two rows are examples from the LSP dataset, and the 3rd row is from the FLIC dataset. The last row shows some failure cases, which are typically due to large foreshortening, occlusions and distractions from clothing or overlapping people.

Conclusion
We have presented a graphical model for human pose which exploits the fact the local image measurements can be used both to detect parts (or joints) and also to predict the spatial relationships between them (Image Dependent Pairwise Relations). These spatial relationships are represented by a mixture model over types of spatial relationships. We use DCNNs to learn conditional probabilities for the presence of parts and their spatial relationships within image patches. Hence our model combines the representational flexibility of graphical models with the efficiency and statistical power of DCNNs. Our method outperforms the state of the art methods on the LSP and FLIC datasets and also performs very well on the Buffy dataset without any training.