Computer Science
Computer Science-2021.gz:143179
Computer Science
b95b3d49086ec84df6d81075fbaf502292046132
FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integration

Most existing monocular 3D pose estimation approaches only focus on a single body part, neglecting the fact that the essential nuance of human motion is conveyed through a concert of subtle movements of face, hands, and body. In this paper, we present FrankMocap, a fast and accurate whole-body 3D pose estimation system that can produce 3D face, hands, and body simultaneously from in-the-wild monocular images. The core idea of FrankMocap is its modular design: We first run 3D pose regression methods for face, hands, and body independently, followed by composing the regression outputs via an integration module. The separate regression modules allow us to take full advantage of their state-of-the-art performances without compromising the original accuracy and reliability in practice. We develop three different integration modules that trade off between latency and accuracy. All of them are capable of providing simple yet effective solutions to unify the separate outputs into seamless whole-body pose estimation results. We quantitatively and qualitatively demonstrate that our modularized system outperforms both the optimization-based and end-to-end methods of estimating whole-body pose.


Introduction
Estimating 3D human pose from single RGB images is one of the core technologies to build a computational model to understand human behavioral cues. It can faciliate numerous applications including assistive technology [23,41,53], sign language understanding [10], AR/VR [3,68], and social signal understanding [51]. Importantly, the essential nuance of human behaviors is conveyed through a concert of subtlest movements of face, hands, and body. Thus, it is necessary to estimate whole-body motions to capture the authentic signals.
Estimating whole-body 3D poses including a face and hands, however, remains challenging. One of the major difficulties comes from the fact that the scale of faces and hands is much smaller than that of torsos and limbs. Furthermore, hands in motion are susceptible to artifacts caused by abrupt viewpoint changes, self-occlusions, and motion blur. These factors largely raise the difficulties of creating large-scale datasets with whole-body 3D poses, even in a controlled environment [32,40,56], not to mention capture such data in the wild. The lack of whole-body 3D data is a major obstacle preventing the construction of a unified system to estimate whole-body 3D poses simultaneously. Therefore, most existing methods focus on one of the major body parts individually: single image-based estimation of 3D body pose (i.e., torso and limbs) [5,33,37,38,60,69,70], 3D hand pose [8,9,20,28,75,78] or 3D face [12,14,19,61,65].
In this paper, we present FrankMocap, a modular system for estimating whole-body 3D poses in a unified output in SMPL-X form [56], as visualized in Fig. 1. Our system is built upon the insight that training a single model to jointly estimate whole parts is intrinsically limited by the lack of accurate and diverse whole-body motion data. Instead, we design a modularized system to first run 3D pose regression methods for face, hands, and body independently. The three regression outputs are then composed via an integration module. The separate regression modules allow us to take full advantage of their state-of-the-art performances without compromising the original accuracy and reliability in practice. Our integration module provides a simple yet effective solution to unify them into seamless whole-body pose estimation outputs. Unlike previous whole-body pose estimation methods that require computationally heavy optimization [32,56,70], our method also allows us to run at an interactive frame rate. We quantitatively and qualitatively demonstrate that our modular framework outperforms existing optimization-based methods [56,70] and end-to-end models trained to produce whole-body outputs jointly [18].
In conclusion, we make three major contributions. First, towards a practical system for whole-body pose estimation, we present the idea of regression-and-integration to take full advantage of existing single-part pose datasets and circumvent the hurdle of lacking whole-body pose datasets. Especially, our hand module is comparable with SOTA hand-only methods on in-the-wild scenarios. Second, we present three effective strategies to integrate outputs from separate single-part pose estimation modules, namely the fastest "copy-paste" approach, a full-optimization approach, and an intermediate approach via a simple integration network. Third, we quantitatively and qualitatively demonstrate that our modular system outperforms the alternative approaches based on optimization-based methods or end-to-end methods training all parts jointly.

Related Work
3D Parametric Human Body Models. 3D parametric human models are widely used as a strong prior to estimate 3D pose and shape of humans. A main idea is to model the deformation of 3D human (including face, hands and body) via low dimension parameters [1,32,[56][57][58]. SCAPE [1] is a pioneering work that accounts for shape variations and pose deformations of human body. Loper et al. introduce SMPL [46] that learns local pose-dependent blendshape on top of linear blend skinning for holistic mesh deformation as well as shape variations. Similarly, there exists a hand deformation model called MANO [58] and several face models [4,11,43]. Recent approaches also introduce whole-body models such as Adam [32], SMPL-X [56], and GHUM & GHUML [71] that are capable of expressing face, body, and hands in a unified space. Single Image 3D Body, Hand and Face Estimation. Recent monocular 3D body motion capture approaches adopt parametric model such as the SMPL [46] or Adam model [32,70] for 3D body representation 1 . A pioneering work [5] uses an optimization framework to fit the 3D body model to 2D observations. More recent methods [21,24,30,33,34,37,42,44,49,52,59,66,72] leverage a deep learning framework to regress parameters of the body models from RGB images. Non-parametric methods have been also introduced by directly regressing the model vertices [17,38] or corresponding UV maps [73]. There are also approaches that use a hybrid framework by using a deep learning framework to produce intermediate representations such as 2D heatmaps and then fitting the skeletal models on these outputs to reconstruct joint angles [47,48,70]. Due to the lack of in-the-wild training data with 3D annotations, most of these models are trained with mixed datasets, including indoor datasets such as Hu-man3.6M [27] and in-the-wild datasets such as COCO [45]. While most methods are based on single images as input, recent approaches [34,36] predict 3D motions from a video.
Monocular 3D hand pose estimation approaches share a similar pipeline as the body counterparts, based on the parametric 3D hand models, such as MANO [58]. Deep neural networks are leveraged to either predict model parameters [2,8,15,16,75,77] or directly regress hand mesh vertices [20,39]. Similarly, single image 3D face prediction methods [14,19,61,65] also share similar ideas by leveraging deep neural networks to regress the face landmarks or parameters of face models, such as 3DMM [4] or FLAME [43]. We refer the readers to recent surveys on 3D face reconstruction [50,80] for more details. Joint 3D Estimation of Body, Hands and Face. There are a few methods [18,56,70,76] aiming to capture wholebody 3D motions. SMPLify-X optimizes the parameter of the SMPL-X model [56] to fit it to 2D keypoints with additional constraints, including body pose priors and collision penalizer. Monocular Total Capture (MTC) [70] is based on the Adam model [32], and adopts deep neural networks to get 2.5D predictions first. Then the parameters of Adam are obtained through optimization. Both of these methods rely on optimization procedures with relatively slow computational time (from 10 seconds to a few minutes). Zhou et al. [76] use SMPL-H [58] to represent body and hands and 3DMM face model [64] to represent 3D faces. They first predict 3D body and hand poses and then separately predict the parameters of 3DMM model, including shape, expression, albedo, and illumination. The predicted full-body motion is not represented in a unified format and this paper does not present generalization ability to in-the-wild scenar-

Body Module
Integration Module

Whole Body Pose
Hand Module

Input Image
Part Poses Figure 2: Overview of FrankMocap, the proposed whole-body 3D pose estimation system. Given a single RGB image input, we first apply our part modules separately to estimate 3D hands, body, and face. Our integration module is then adopted to combine the outputs from each module into a unified whole-body output.
ios. ExPose [18], avoids using the optimization procedure by presenting a neural network to simultaneously predict the body, hand, and face parameters of SMPL-X. In particular, ExPose curates a pseudo-ground truth dataset by fitting SMPL-X model on in-the-wild images, followed by manual quality checking by annotators, and use it to train their model to jointly produces output.

Method
Given a single image as input, FrankMocap firstly estimates the 3D poses of face, hands (both left and right), and body (the torso and limb parts), in the form of SMPL-X model [56]. Each part's output is produced by a separate 3D pose regressor trained on public datasets in each sub-field. Their outputs are combined through our integration module, to produce a seamless and unified whole-body pose estimation output. Fig. 2 illustrates an overview of the proposed framework.

Whole-Body Parameterization
SMPL-X Model. We formulate the SMPL-X model as: where W is parameterized by global orientation of the whole-body φ w ∈ R 3 , facial expressions parameters ψ f ∈ R 10 , whole-body pose parameters θ w ∈ R (21+15+15+1)×3 accounting for pose-dependent deformation, and shape parameters β w ∈ R 10 accounting for cross-identity shape variations of the face, hands, and body. We divide θ w for each of the body, hands, and faces, namely body pose parameters θ b w ∈ R 21×3 , left hand pose parameters θ lh w ∈ R 15×3 , right hand pose parameters θ rh w ∈ R 15×3 , and face pose parameters 2 θ f w ∈ R 1×3 . In this way, All pose parameters are defined in the axis-angle representation, which stores the relative rotation to the parent joints defined in the kinematics map. As output, the SMPL-X model produces a mesh structure with 10,745 vertices, V w ∈ R 10475×3 . The 3D joint locations of the whole-body can be obtained by applying a joint regression function R from the posed vertices as . Stand-Alone Hand Model. Our hand model is defined by taking the hand parts of SMPL-X: where θ h ∈ R 3×15 is hand pose parameters and β h is the shape parameters for the hand model. Since our hand model is taken from SMPL-X, β h shares the same shape space as β w . For brevity, we use θ h , instead of θ rh or θ lh , to denote the hand pose parameters of either part in our description. φ h ∈ R 3 represents the global orientation of the hand meshes. Our hand model H produces the hand mesh structure with 778 vertices, V h ∈ R 778×3 , where the hand vertices are selected from the hand area of the original SMPL-X vertices. Given hand vertices V h , 3D hand joints can be regressed as contains a wrist, 15 finger joints (3 joints per finger), and 5 finger tips. Hand visualization and skeleton hierarchy are listed in Appendix A1.1. The major advantage of our hand representation is that the components of this 3D hand model, including pose parameters, vertices, and 3D joints, are directly compatible with the whole-body parameterization. This enables us to efficiently integrate outputs from the body module and the hand module.  Figure 3: Overview of hand module. Our hand module takes a cropped hand image IH as input, and produces the parameters of hand model,

y y 5 w T N h f H 7 B 6 z S s V g = < / l a t e x i t >
The estimated parameters are then used to calculate the mesh of SMPL-X hand part.

3D Hand Estimation Module
We present a monocular 3D hand pose estimation module, denoted by M H , to the parameters of the hand model H. In particular, our hand module is inspired by the recently proposed monocular body pose estimation approaches [33,34,37]. We follow similar model architecture, parameterization, and training strategies as the pioneering body methods. In particular, we introduce blur augmentation to make the hand module more robust to blurry hands that are frequently observed in full body motion capture scenarios. Network Architecture. Our hand module M H is built upon an end-to-end deep neural network architecture that can predict hand model parameters defined in Eq. (2) from given input images. This process is defined as: where I H is an input RGB image cropped around the hand region. c h = (t h , s h ) is a set of weak-perspective camera parameters to project a posed 3D hand model to the input image. t h ∈ R 2 is for 2D translation on the image plane, and s h ∈ R is a scale factor. Given camera parameters, the i-th 3D hand joint, J 3D h,i can be projected as: where Π is an orthographic projection. Following the practice of HMR [33,37], our hand module M H adopts an encoder-decoder structure, where the encoder (ResNet-50 [26]) outputs features from an input image, and the decoder regresses the hand model parameters from the features. See Fig. 3 for the overview of our hand module. In particular, our hand module is trained with the assumption that the inputs are right hands. The images and annotations for the left hand are used after vertical flipping. During inference, the left-hand images are first flipped and processed as if they were a right hand. Then the outputs are flipped back to the original left-hand space. Note that the shape parameter β h is originally defined for whole-body model β w , but we only consider the deformation for the hand vertices defined in Eq. 2D keypoint annotations. The losses for each of the annotation, namely L θ , L 3D and L 2D , are defined as follows: are the ground-truth axis-angle pose parameters, 3D keypoints, and 2D keypoints, respectively. In particular, the 2D keypoint loss are leveraged to estimate the camera parameters and facilitate the hand module generalize to in-the-wild images with only 2D keypoints annotation. We do not use the shape parameters provided by the 3D hand datasets such as FreiHAND [79], since these are defined for the MANO model [58] and not compatible with our hand model from SMPL-X. Instead, an additional shape parameter regularization loss L reg = β h 2 2 is applied. The overall loss L used to train our hand module is defined as follows: In experiments, the balanced weights are set as λ 1 = 10, λ 2 = 100, λ 3 = 10 and λ 4 = 0.1. Other hand module training details, including data preprocessing, dataset information, and implementation details, are included in the Appendix A1.1. Motion Blur Augmentation. Performing data augmentations during training is a common practice to enable a model with better generalizability. We firstly apply common data augmentation strategies, including random scale, random translation, color jittering, and random rotation. Importantly, we recognize that in-the-wild videos are often accompanied by severe motion blur. To achieve robustness to motion blur, we additionally apply motion blur augmentation to the images. We use the methods [6,7] to generate blur kernels and then use 2D filtering to add blurriness to images. Examples of applying motion blur augmentation are shown in Fig. 4. Our experiments clearly show that our motion blur augmentation can effectively insist our hand module generalize to in-the-wild scenarios.

3D Body Estimation Module
Our body module M B produces the torso and limb parameters defined in Eq. (1) from a single image: where I b is an input image cropped around a target single person's whole-body. Similar to Eq. (1), φ b ∈ R 3 is the global body orientation, θ b ∈ R 21×3 is the body pose parameters (without any finger joints), and β b ∈ R 10 is the shape parameter. β b shares the same parameterization space as β w , as defined in Eq. (1). Similar to Eq. (3), we use weak perspective camera parameters We leverage the publicly available body pose estimation model (SPIN [37]) trained by the EFT dataset [30].
Since the original SPIN model is built with SMPL, we finetune the network by replacing SMPL with SMPL-X. For the fine-tuning process, we use the existing indoor 3D pose dataset Human3.6M [27] and outdoor pseudo-GT dataset by EFT [30] that provides SMPL annotations. Since the annotation (SMPL) and our target model (SMPL-X) are not exactly compatible, we only use the pose parameters and 2D keypoint from the annotations during fine-tuning process, ignoring the shape parameters 3 .

3D Face Estimation Module
We use the off-the-shelf public 3D face estimation model RingNet [61] to estimate facial expressions ψ f and face poses θ f as follows: Since RingNet is based on FLAME [43], its predictions are compatible with the face part of SMPL-X. As an adjustment, we only keep the first 10 expression parameters from the original 50 dimensional expression parametersψ f predicted by RingNet, to make it compatible with the expression space defined in SMPL-X. It is worth noting that the FLAME and SMPL-X share the same PCA-based expression space, while SMPL-X only uses the 10 foremost principal components.

Whole-Body Integration Module
Our integration module combines the outputs from the face, hands, and body modules into a unified representation of the SMPL-X [56]. While the output of the face module can be easily applied to the face part of the SMPL-X model by copying the expression parameters and jaw poses, additional procedures are required to integrate the outputs of the hand and body modules because different wrist poses are estimated from these two modules. For this objective, we present three strategies: (1) by simple copy-paste composition for fastest processing, (2) by an optimization framework, (3) by an integration network to approximate the optimization via a simple neural network. By Copy-Paste. Our hand and body modules' outputs can be efficiently combined, since they are both defined in SMPL-X parameterization. As the simplest strategy, we transfer the corresponding joint angle parameters from the outputs of each module into the whole-body model. The wrist pose parameters require additional processing because we obtain two different outputs from the body module and the hand module (the wrist pose of hand module is represented by the global hand orientation φ h ). Let us denote the pose parameters for the wrist joint as θ wrist , then }, whereθ b represents all body pose parameters except wrists. We use the similar notations for the whole-body pose parameters, θ b w =θ w ∪ {θ rwrist w , θ lwrist w }. Then, whole-body integration by copy-paste can be performed as: where Γ {l,r} are the functions to convert the global wrist orientation φ h obtained from the hand module to the local wrist pose parameters w.r.t. its parent joint in the SMPL-X skeleton hierarchy. This can be implemented by comparing φ h with the global orientation of the current wrist pose from θ b that can be computed by following the forward kinematics of the body skeleton hierarchy. This strategy requires almost no extra computation, making our separate modules contribute a common whole-body model simultaneously. We found this simple integration produces convincing results, especially for the scenarios with computational bottlenecks as in our live demo. By Optimization. As an alternative integration method for better accuracy, we build an optimization framework to fit the whole-body model parameters given the outputs from body and hand modules. This strategy is particularly helpful to reduce the artifact around the wrist parts over the copy-paste strategy. It can also take advantage from the 2D keypoint estimation output [13] for better 2D localization quality. Our optimization framework finds the wholebody model parameters that minimize the following objective cost function: where F 2d is the 2D re-projection cost term between the 2D keypoint estimation [13] and the projection of 3D joints (hand, body and faces). F mesh is the 3D distance loss between the hand mesh of whole-body model and the mesh vertices of the hand module's output localized on the body's wrist joints, as shown in Fig. 5. The prior term F pri is needed to keep the 3D pose and shape parameters in plausible space. We use the EFT introduced in [30] for the similar goal, by applying neural network finetuning to replace explicit 3D pose prior terms used by previous works [56]. Therefore, the prior term only accounts for the shape regularization F pri = β w  We use a multi-stage approach in which optimization is firstly performed without F mesh . Then we optimize all terms in the second stage. During the second stage, we add an additional constraint F 3D to maintain the 3D joints within the final locations of the first stage. Our experiments show that this term can produce reliable results in practice. See Fig. 5 for the example of our optimization. By Wrist Integration Network. The result of the hand module shows precise 2D localization quality as shown in Fig. 6. But we lose its precision by applying copy-paste integration strategy since the wrist joint location is determined by the body module, as shown in Fig. 5 (second last  column). Although the optimization method can achieve better precision and resolve this problem, it suffers from relatively slow computation speed due to the requirement of iterative gradient computations. As another alternative strategy to achieve both precision and fast-runtime performance, we introduce an integration network to adjust the arm pose of the whole-body model toward the given 2D location without gradient computation. Specifically, we develop a small neural network to achieve this: where d ∈ R 2 is a 2D directional vector that the arm needs to follow in the image space. It is obtained from the wrist locations calculated from the body module and those from the hand module, followed by a normalization with the length of the arms. θ arm w is the pose parameters of the elbow and shoulder joints after the copy-paste integration. Conceptually, H predicts an approximation of optimization gradient to adjust the arm parameters, similar to the recent work of Song et al. [63]. We use a MLP with six layers to implement H. To train H, we generate a synthetic dataset by capturing our own a range of motion videos to cover diverse arm pose variations and by simulating arbitrary d directions. More details of wrist integration are described in Appendix A2.

Experiments
We first demonstrate that our hand module achieves high performance competing to the previous state-of-the-art 3D hand methods, along with ablation studies to examine the designs of our method. Then, we demonstrate that our whole-body pose estimation method outperforms previous approaches on a public benchmark. See Appendix A1 for the details of implementation and datasets.

Hand Module Evaluation
Comparison with State-of-the-art Methods. We compare our hand module with the previous state-of-the-art hand approaches on three public hand benchmarks, namely STB [74], RHD [78] and MPII+NZSL [62]. The results are listed in Tab. 1 and Fig. 6. For each validation dataset, we  Fig. 7 further show that adopting data augmentation helps our model generalize better on challenging scenarios, including blur, pose variations, and occlusion. In Fig. 7, "No Augment" refers to the model trained without any data augmentation, and "No Blur" refers to the model trained with all data augmentation strategies except the motion blur augmentation. "Full" refers to the model trained with all data augmentation strategies. We also do an ablation study on mixture usage of datasets, demonstrating that using diverse datasets improves the performances. See Appendix A1.3 for details on this part.

Body Module Evaluation
We compare our body module with the previous SOTA body-only methods on the test set of 3DPW [67]. Following the previous approaches, we use Mean Per Joint Position Errors (MPJPE) and PA-MPJPE (after Procrustes analysis [22]). The results are listed in Tab. 3. Our body module is comparable to the previous state-of-the-arts, including ExPose [18]. Our body model is a fine-tuned version of SPIN [37] and EFT [30]. The slightly degraded performance might be due to the reason that we only use fewer datasets, COCO and Human3.6M datasets. We only compare with HMR [33]-based methods. We believe that the experimental results shown in Tab. 3 can demonsrate the feasibility of extending previous SMPL-based methods to SMPL-X based body module. Thanks to our modularized design, we can also easily leverage the most recent 3D body  poses methods [42,44] to build our body module.

Integration Module Evaluation
We quantitatively compare our methods with previous SMPL-X based whole-body methods (SMPLify-X [56] and ExPose [18]) on EHF dataset [56]. The evaluation metrics are vertex-to-vertex distance (V2V) and Procrustes Analysis vertex-to-vertex distance (PA-V2V). The results are listed in Tab. 4. We run the officially released codes of SMPLify-X and ExPose to obtain the evaluation results. FM-CP, FM-WI, and FM-OPT indicate FrankMocap using the copy-paste, wrist integration network, and optimizationbased as the integration module, respectively. The results demonstrate that all our methods significantly outperform the previous methods in terms of whole-body estimation, body estimation, and hand estimation. Our face estimation is comparable with previous methods, where we only use an off-the-shelf 3D face model [61]. Comparing across different versions of FrankMocap reveal that the model with optimization achieves the best performance in general. Applying our wrist integration network can also lead to better performance than copy-paste version. We also provide qualitative comparisons between different integration modules in Fig. 8. It is revealed that using the wrist integration network can bring better hand location while adopting optimization can further improve hand localization while updating body shape and other poses.
We also qualitatively compare our method with previ- Figure 9: Qualitative comparison over the previous whole-body pose estimation methods. We compare our method adopting the wrist integration module (FM-WI) with MTC [70], SMPLify-X [56] and ExPose [18].
ous whole-body motion capture approaches, MTC [70], SMPLify-X [56], and ExPose [18], with our output with wrist integration network. Results shown in Fig. 9 and the demo video demonstrate that our method outperforms previous approaches, with fast and accurate performance in challenging in-the-wild scenes. Qualitative comparison with Zhou et al. [76] is included in the Appendix A5.

Conclusion
We present FrankMocap, a monocular 3D whole-body motion capture system built upon the motivation to take full advantages of state-of-the-art part estimation modules, followed by effective integration algorithms. We develop three integration modules, namely copy-paste, wrist integration network, and an optimization-based method. The copypaste and wrist integration modules are suitable for timesensitive applications, while the optimization-based module is suitable for offline applications requiring better precision. Our system surpasses previous state-of-the-art 3D wholebody estimation methods on both public benchmarks and in-the-wild scenarios. Notably, our approach can directly leverage future algorithms by replacing each module, which is a main advantage of our modular design.
In this section, we provide more details of our standalone hand module described in Sec. 3.2. We first describe the hand mesh and hierarchy of the hand skeleton. Then we discuss how to preprocess data from different datasets. Then we discuss the implementation details of our hand module. In the end, we discuss the hand datasets used in our paper and provide experiment results to demonstrate the effectiveness of adopting different datasets.

A1.1. Hand Module Details
Hand Mesh. The visualization of cropped hand mesh and hierarchy of the hand skeleton is depicted in Fig. 10. Data preprocessing. Following the practice of 3D body pose estimation methods [33,34,37,59], we include as many publicly available datasets as possible towards inthe-wild generalization ability. To handle the discrepancy of annotations across different datasets, we perform several pre-processing steps to make them consistent and compatible with our hand model. The pre-processing includes: (1) Rescaling all 3D keypoint annotations to be compatible with our hand model, by using the middle finger's knuckle length as a reference. (2) Reordering the 3D keypoints to be the same as our hand model's skeleton hierarchy. Hand Module Implementation Details. Input images of the hand module are center-cropped surrounding the hands, where the bounding boxes for cropping are given by 2D hand keypoints. We use ground-truth 2D keypoints during training and predicted keypoints from OpenPose [13] during testing. The cropped images are further padded and resized to 224 × 224. During training, we apply data augmentations to each of the training images via random scaling, translation, rotation, color jittering, and synthetic motion blur. The hand module architecture is based on ResNet-50 [26] with two additional fully connected layers to map the output features of ResNet to vectors with 61 dimension, which is composed of camera parameters c h (3 dimensions), hand global rotation φ h (3 dimensions), hand pose parameters θ h (45 dimensions) and shape parameters β h (10 dimensions). The hand module is implemented with PyTorch [55]. The Adam optimizer [35] with learning rate 1e−4 is used to train the model. The hand module is trained until converge, which takes about 100 epochs.

A1.2. Hand Datasets
To increase our hand module's generalization ability, we adopt as many public datasets as possible. The summary of different datasets is listed in Tab. 5. The details of these datasets are listed below. FreiHAND. FreiHAND [79] is a dataset with ground truth 3D hand joints and MANO parameters for real human hand   [79], HO-3D [25], MTC [70], STB [74] and RHD [78]. We use MPII+NZSL [62] for validation only.
Information → # of Samples Pose Angles 3D Joints 2D Joints Dataset ↓ FreiHAND [79] 60K HO-3D [25] 60K MTC [70] 200K STB [74] 20K RHD [78] 50K MPII+NZSL [62] 10K images. The 3D annotations are obtained by a multi-camera system and a semi-automated approach. The obtained data is further augmented with synthetic backgrounds. In our experiments, we randomly select 80% of samples from the original training set as training data and use the remaining 20% of samples for validation. HO-3D. HO-3D dataset [25] is a dataset aiming to study the interaction between hands and objects. The dataset has 3D joints and MANO pose parameters for hands. It also has 3D bounding boxes for objects the hands interact with.
In this paper, we only use 3D annotations of hands. The training set is composed of different sequences, each of which records one type of hand-object interaction. Following the similar practice in processing FreiHAND, we randomly choose 80% of sequences from the original training set as training data and use the remaining 20% of sequences for validation. MTC. Monocular Total Capture [70] is a dataset captured by Panoptic Studio [29,31] in a multi-view setup with 30 HD cameras. It has 3D hand joints annotations for both body and hands. The sequences are mainly the range of motion data of multiple subjects. To polish the dataset, we filter out erroneous samples where hands are not visible or too small. STB. Stereo Hand Pose Tracking Benchmark [74] is com- Figure 11: Ablation study on training datasets. We show qualitative ablation study on using different datasets in training our hand model. "Subset-01" means using the combination of datasets FreiHADN [79] and HO-3D [25]. "Subset-02" means using the combination of datasets: STB [74], RHD [78] and MTC [70]. "Full set" means using all the above datasets. The images are selected from COCO dataset [45].

A1.3. Ablation Study on Hand Datasets.
We examine the influence of using different datasets in this subsection. The results are listed in Tab. 6 and Fig. 11. As expected, the results in Tab. 6 show that using more datasets will lead to better performance. We also show examples of qualitative comparison in Fig. 11. Similar to the conclusion from the quantitative study, the qualitative results show that incorporating more datasets can increase the models' generalization ability and generate more precise results for in-the-wild images. In the figure, "Subset-01" means using the combination of datasets FreiHAND [79] and HO-3D [25]. "Subset-02" means using the combination of datasets: STB [74], RHD [78] and MTC [70]. "Full set" means using all the datasets.

A2.1. Framework Details.
The framework of wrist integration network is depicted in Fig. 12. Given hand prediction from hand module and whole-body prediction from copy-paste integration module, the role of wrist integration network is to adjust the arm poses to move hands to the locations determined by the hand module. The input of the wrist integration network is composed of two parts. The first part is the 2D directional vector d that the arm needs to follow in the image space. It is obtained from the wrist locations calculated from the body module and those from the hand module, followed by a normalization with the length of arm. The second part θ arm w is the pose parameters of the elbow and shoulder joints the copy-paste integration. The original shoulder poses θ s w and elbow poses θ e w are all local rotations relative to their parent defined in the kinematics of body skeleton. To ease the training of the wrist integration network, we transform the shoulder poses to the global orientation φ s w , so that the wrist integration network does not need to consider other body parts' poses. The elbow poses are kept as the local rotation. In this way, the input arm poses θ arm w is composed of global orientation of shoulder φ s w and local rotations of elbow θ e w .

Origin Prediction
Wrist Integrate WI Input WI Output Figure 12: Overview of the proposed wrist integration network.
where Υ s is the function to convert the local shoulder rotation θ s w to global shoulder orientation φ s w . Υ s can be implemented by forwarding the kinematics of the SMPL-X body skeleton from the root to the shoulder.
Taking the arm poses θ arm w and 2D directional vector as input, the wrist integration network predicts the updated arm posesθ arm w . The whole process is formulated as Eq. (12).θ It is worth nothing thatθ with pose shoulder and elbow arms in the local rotation, as defined in Eq. 10, can be formulated as: where Γ s is the function to convert the global shoulder orientationφ s w to local shoulder rotationθ s w . Its definition is similar to Γ l and Γ r defined in Eq. (8).

A2.2. Implementation Details.
The wrist integration network is a MLP with 6 layers. The input dimension is 8, which is composed of θ arm w ∈ R 2×3 and d ∈ R 2 . The output dimension is 6 which is θ

A2.3. Training Data.
To train H, we generate a synthetic dataset by capturing our own a range of motion of right arms. We first capture videos with varying arm poses. Then we use the optimization method introduced in Section 3.5 to obtain the pseudo ground-truth 3D pose estimations for those video frames. Suppose for one data sample, we obtain φ w , θ w , β w , c w , which are global rotation, whole-body pose, whole-body shapes and camera parameters. To increase the model's generalization ability to varying body shapes, we add random noise to the obtain body shape β w . Then we use the

Input Prediction
Input Prediction updated body shapes and other obtained parameters to get projected right 2D hand joints J 2D rh by projecting the predicted 3D right hand joints J 3D rh using Eq. 4. After that, we add random translationḋ = [ḋ x ,ḋ y ] to the original predicted 2D hand joints to obtain permutated 2D hand jointṡ J 2D rh = J 2D rh +ḋ. We run the optimization method again to obtain the new poses using the permutated 2D hand jointṡ J 2D rh in calculating F 2d defined in Eq. (9). In this process, we fix other parameters and only update the right shoulder and right elbow poses, obtainingθ arm w . In the end, we obtain training data with hand directional vectorḋ, original poses θ arm w and updated arm posesθ arm w . They form the training data to train wrist integration network defined in Fig. 12.
We visualize several generated data samples in Fig. 13. The right arrows are the random translationḋ to be added. The green bodies are rendered from the original optimization prediction. The blue arms are the obtained arm poses θ arm w after applying randomly translating.

A3. Implementation Details
Bounding Bboxes. We use a lightweight Pytorch Open-Pose implementation [54], which only predicts 2D body keypoints, to obtain body bounding bboxes. Face and hand bounding bboxes can also be obtained from OpenPose [13]. For live demo requiring higher speed, the bounding boxes are obtained by projecting the face and hand part of the estimated 3D body to image space.

A4. Failure Cases
We show several typical failure cases in Fig. 14. It is revealed that failure cases are typically caused by severe occlusion and blurry.

A5. More Qualitative Results
Comparison with Zhou et al. Firstly, we show qualitative comparisons with Zhou et al. [76] in Fig. 15. It is worth Input Zhou et al Ours Figure 15: Quanlitative comparison with Zhou et al. [76]. Hand regions are marked with red circles. It is revealed that our method can generate more precise 3D hand poses than Zhou et al. [76] do.
noting that Zhou et al. [76] have not released their official code yet. Therefore, we compare our methods with them using images shown in their paper. The results demonstrate that our methods can generate more accurate hand and body poses than Zhou et al. [76] does. Demo. We show more qualitative results of FrankMocap in Fig.. 16. The integration module is the Wrist Integration Network. More qualitative results are shown in the demo videos.