3D Keypoint Triangulation.
As the camera intrinsic and extrinsic parameters are available, we triangulate 3D keypoints \(\mathcal {P}_{3D} \in \mathbb {R}^{P \times 3}\)  with the multi-view 2D estimated keypoints \(\hat{\mathcal {P}}_{2D}\) . However, 2D keypoints from any single view may not be always reliable. Hence, we use the following strategies to improve the quality of 3D keypoints.
1) Keypoint selection. To avoid the influence of poor-quality estimated 2D keypoints, we use a threshold \(\tau _k\)  to remove keypoints with a low confidence score.
2) Camera selection. As our system consists of ten Kinects, we exploit the redundancy to remove low-quality views. We only keep camera views with reprojection errors that are top-\(k\)  smallest [1]} and no larger than a threshold \(\tau _c\) .
3) Smoothness constraint. Due to inevitable occlusion in the single view, the estimated 2D keypoints often have jitters. To alleviate the issue, we develop a smoothness loss to minimize the difference between consecutive triangulated 3D keypoints. Note that we design the loss weight to be inversely proportional to average speed, in order to remove jitters without compromising the ability to capture fast body motions.
4) Bone length constraint. As human bone length is constant, the per-frame bone length is constrained towards the median bone length \(\mathcal {B}\)  pre-computed from the initial triangulated 3D keypoints.
The constraints are formulated as Eq. REF :
