MBW: Multi-view Bootstrapping in the Wild

Labeling articulated objects in unconstrained settings have a wide variety of applications including entertainment, neuroscience, psychology, ethology, and many fields of medicine. Large offline labeled datasets do not exist for all but the most common articulated object categories (e.g., humans). Hand labeling these landmarks within a video sequence is a laborious task. Learned landmark detectors can help, but can be error-prone when trained from only a few examples. Multi-camera systems that train fine-grained detectors have shown significant promise in detecting such errors, allowing for self-supervised solutions that only need a small percentage of the video sequence to be hand-labeled. The approach, however, is based on calibrated cameras and rigid geometry, making it expensive, difficult to manage, and impractical in real-world scenarios. In this paper, we address these bottlenecks by combining a non-rigid 3D neural prior with deep flow to obtain high-fidelity landmark estimates from videos with only two or three uncalibrated, handheld cameras. With just a few annotations (representing 1-2% of the frames), we are able to produce 2D results comparable to state-of-the-art fully supervised methods, along with 3D reconstructions that are impossible with other existing approaches. Our Multi-view Bootstrapping in the Wild (MBW) approach demonstrates impressive results on standard human datasets, as well as tigers, cheetahs, fish, colobus monkeys, chimpanzees, and flamingos from videos captured casually in a zoo. We release the codebase for MBW as well as this challenging zoo dataset consisting image frames of tail-end distribution categories with their corresponding 2D, 3D labels generated from minimal human intervention.


Introduction
Hand labeling landmarks of articulated objects within video is an arduous and expensive task. Landmark detectors [34,25,37] can be employed to automate the process. However, they require the ingestion of large amounts of labeled training data to be reliable -an infeasible requirement for all but the most common of articulated objects (e.g. people, hands). Semi-supervision can help [33], where a small portion of frames within the video are hand labeled. Candidate labels can be generated from the noisy landmark detectors -trained from the seed hand labeled examples -inliers are then determined through calibrated rigid multi-view geometry. These inliers are treated as labels and used to train the next round of landmark detectors. This semi-supervised process is iterated to increase the number of inlier estimates, with additional human annotation being added judiciously to ensure the full sequence is labeled. Such strategies have been instrumental for obtaining reliable ground-truthmost notably the Multi-view Bootstrapping (MB) approach of Simon et al. [33]. Human annotators Figure 1: Overview of our MBW approach. Top: Provided an unconstrained Multi-view uncalibrated video with very few 2D labels (≈ 1 − 2 % or ≈ 15 labels), our method recovers the 3D structure in a canonical frame, along with camera poses and corresponding 2D landmarks for the complete video sequence. Bottom: Diverse reconstructions and data labeling for videos captured in the wild. This dataset is released as part of the paper.
are only required to hand label a subset of the dataset, with the rest just requiring visual inspection to validate the accuracy of the inferred labels.
Although significantly cutting down on human labor, Multi-view Bootstrapping [33] is still expensive and cumbersome, requiring a static multi-camera rig which usually consists of tens [1] or even sometimes hundreds of calibrated cameras [14]. The number of cameras can be reduced, but with a trade-off in decreasing robustness of outlier rejection and increasing human interventions (see Fig. 4). This makes it less feasible for capturing objects outside laboratories. In this paper, we advocate for a significant advancement by enabling its application to data captured by a few (2 to 4) handheld cameras with only a handful of annotated frames (about 10-15 frames per several minutes of video). We refer to our approach herein as Multi-view Bootstrapping in the Wild (MBW). The cameras need not be calibrated, and fields of view need only overlap the articulated object, not the backgrounds.
Our innovations come from (i) utilizing Multi-View Non-Rigid Structure from Motion (MV-NRSfM) [3] to more reliably estimate camera poses and 3D landmark positions from noisy 2D inputs with few cameras. Compared to performing SfM / triangulation independently for each frame as in prior works [14,1], MV-NRSfM leverages the redundancy in shape variations among different frames, thus it is less sensitive to the variations of input views, more capable of detecting outliers and denoising inlier 2D landmark estimates. (ii) We leverage recent advances in deep optical flow [35] as an alternative strategy for creating landmark label candidates -something especially useful in the early iterations of the semi-supervision process.
As a result our approach can be effectively applied to less studied articulated object categories. We show results on tigers, fish, colobus monkeys, gorillas, chimpanzees, and flamingos from a zoo dataset (captured by the authors, who hereby release it under a CC-BY-NC license). We also quantitatively evaluate the proposed pipeline on common motion capture datasets (e.g. Human3.6 Million [12]). The accuracy of the learned landmark detector is competitive to state-of-the-art fully supervised method. A graphical depiction of our approach can be found in Figure 1.

Related Works
Panoptic Studio [14] paved the way for collecting data for deformable objects such as the human body. Subsequent efforts on humans [12,29,41], hands [43,44,24], monkeys [1], canines [17], chee-  [27] No N/A No No 5% Simon et al. [33] No Required Yes No 30% MBW (Ours) Yes No Yes Yes 2% Figure 2: (Dotted lines) The MV-NRSfM neural shape prior is initially trained with labels for 1-2% of the frames (shown as green images). A pre-trained optical flow network then propagates the initial labels through the video to generate additional 2D candidates. Candidates that result in high reprojection error from the 3D lifting network are rejected as outliers (red). (Solid line) From here on, the label set is updated with inliers from the previous iteration, and is then used both to retrain the MV-NRSfM and to train a 2D detector. Dotted line is executed only once while solid lines are repeated for K iterations.
tahs [15], rats [20], and insects [8] have followed. Multi-view Boostrapping [33] has demonstrated how these calibrated multi-camera datasets can be labeled efficiently through a semi-supervised learning paradigm and a small number of hand annotations. A fundamental drawback to multi-view bootstrapping however is that it requires a large number of views and accurate camera calibration.
Recent works have explored alternate paradigms for semi-supervised landmark labeling that do not require such exotic calibrated multi-camera setups. Mathis et al. [22], Pereira et al. [27], and Yu et al. [40] tackle this problem from a single view, but largely ignore the use of multi-view geometry. Gunel et al. [8] have explored an approach that utilizes a small number of camera views, and only requires an approximate estimate of the camera extrinsics. They use pictorial structures [5] to automatically detect and correct labeling errors, and use active learning to iteratively improve landmark detection performance. Although this approach is useful in lab settings where there are static cameras and the object is anchored to a fixed location (e.g. tethered flies are positioned over a spherical treadmill [8]), it is non-trivial to generalize such performance to more complex environments and across significant individual variations due to e.g. patterned skins in animals or demographics and clothing in humans. In contrast, our approach accepts image frames from moving cameras and requires only a handful of hand annotated labels. Further, it does not require any camera information, and can easily be applied to a broad set of articulated objects such as humans, hands, and animals. Thus, the strength of our method is its generalizability. Since the provided implementation of DeepFly3D was specific for Drosophila, it was not readily applicable to our in-the-wild datasets. An overview highlighting major differences between our proposed approach and related works trying to achieve a similar application is shown in Tab. 1. Figure 3: Sample sequences composited from our Zoo data collection -situations where traditional SLAM pipelines fail to recover reasonable camera matrices due to lack of reliable matching features.

Problem Setup
Our goal is to learn 2D landmarks of articulated objects from multi-view synchronized videos captured in the wild. Unlike other works [14,8,4,42] developed for laboratory settings, we focus on the in the wild setting, i.e. data is captured using a small number (2 or 3) of cameras with unknown extrinsics, and only a small portion (1 to 2%) of the data is manually labeled.
More specifically, our training set S consists of V synchronized videos, each with N frames. Each training image is denoted as I (n,v) where n ∈ [1, . . . , N ] and v ∈ [1, . . . , V ] denote frame and view indices. Initially only a subset of frames (n, v) ∈ S 0 are given with 2D landmark annotations W (n,v) ∈ R P ×2 of P points. Each row of W (n,v) corresponds to the 2D location of a landmark (e.g. the left knee of flamingo, see Fig. 3). To simplify explanations, we assume that only a single object of interest is visible in each frame. For multiple non-overlapping objects, our algorithm is able to estimate bounding boxes to reduce the problem into a single object case (see Appendix D). Finally, the goal is to (i) infer the missing 2D landmark annotations in the training set as a self-labeling task; (ii) train a 2D landmark detector for unseen objects of the same category.

Learnable geometric supervised self-training
We employ a self-training approach which iteratively assigns pseudo labels and retrains a 2D landmark detector. At each iteration, the 2D pseudo labels generated by a landmark detector are verified using geometric constraints. Samples which fail the verification are dropped, and the remaining pseudo labels are denoised before feeding them back as labels to retrain the landmark detector. Such geometric supervised self-training strategy has been widely used in learning landmark detections [8,14,1,33], what differentiates our work is that we model the geometric constraints as a learnable function, which is learned together with the landmark detector. We abstract this function as: whereW v ∈ R P ×2 represents detected 2D landmarks at v-th view, and y v is the measured uncertainty for outlier rejection. We derive g from performing multi-view non-rigid structure from motion (MV-NRSfM) as described in Sec. 3.3. The remaining details of the self-training pipeline is given as follows.
Initialization In the initial step, we require human labelers to annotate the 2D landmark positions of the same target object for a small portion of captured video frames. We then train our geometric constraint function g using these initial labels. Since the initial labels only cover a limited range of shape variations, the learned g is aggressive in detecting outliers at the beginning stage of the training. It will be improved as it sees more shape variation in each iteration.
Label propagation through tracking We find that directly training a 2D landmark detector such as HRNet [34] using very few labeled samples yields unstable results. To increase the number of training samples, we propagate the annotated 2D landmark labels to the rest of unlabeled frames through tracking. We use an off-the-shelf optical flow network [35] to track the landmarks frame to frame. Other tracking methods [30,9] can also be utilized. We employ standard forward&backward flow consistency check to detect tracking failures. Since the optical flow network tends to make consistent wrong estimations when swapping the input frames, such consistency check alone is not enough to exclude all tracking failures. Therefore, we further employ the learned geometric constraint function g to aggressively remove any likely outliers if the predicted uncertainty y is above a certain threshold. We then add the remaining tracked points (inliers) to the labeled set. This new set is then used both to re-train g, and to train the first iteration of the 2D landmark detector used in the subsequent stages.
Self-training iterations At each iteration t, we define a "labeled" set S t−1 which includes all frames that are either manually annotated, or are labeled by the landmark detector f t−1 in the previous stage and passes outlier rejection using g t−1 . We then re-train the landmark detector and the geometric constraint function on the labeled set S t−1 , which leads to a new detector f t as well as g t . Once trained, inference is run with this detector network f t over all the captured frames. This produces new pseudo labelsW t n,v for all the N frames and V views. We then apply the geometric constraint function g t to evaluate the uncertainty score y t n,v for each pseudo label. Finally we define a new labeled set S t which includes all samples (n, v) that satisfy y t n,v is below a certain threshold. The above process is repeated for a number of iterations. In principle frames that are still not annotated (rejected by g t ) can be actively labeled by humans, however in practice we have found this situation is rare, unless the distance between the captured views is extremely small, making it difficult to learn a reasonable 3D shape prior.

Outlier detection using multi-view NRSfM network
Uncertainty score. Our geometric constraint function g is built upon measuring the discrepancy between detected 2D landmarks and the 3D reconstruction by a multi-view NRSfM method. This is in the same spirit as using the reprojection error of triangulation to measure uncertainties as in prior works. The idea is if the detected 2D landmarks at different views are all correct, we should be able to recover accurate camera poses and 3D structures, and consequently the reprojection of recovered 3D landmarks matches the 2D landmarks. On the other hand, if the reprojection error is high, it means there exists errors in the 2D landmarks which prevents perfect 3D reconstructions. This leads to the following formulation of our uncertainty score, whereT (n,v) ,S n are the estimated camera extrinsics and 3D landmark positions in the world coordinate,W (n,v) is 2D landmarks estimated by the landmark detector, and proj is the projection function. The effectiveness of the uncertainty score defined by Eq. 2 depends on the reliability of estimatingT (n,v) ,S n . However, due to the low number of synchronized views as well as noise inW (n,v) , simply performing SfM and triangulation gives poor result as shown in Fig. 4a. This motivates the following use of MV-NRSfM.
Unsupervised learned MV-NRSfM. Our solution to reliably estimateT (n,v) ,S n is to marry both the multi-view geometric constraints and the temporal redundancies across frames, which leads to the adaption of the MV-NRSfM method [3]. Limited by space, we refer interested reader to their paper for detailed treatment. Here we briefly discuss its usage in our problem. In a nutshell, MV-NRSfM [3] assumes that 3D shapes (concatenation of 3D landmark positions) can be compressed into low-dimensional latent codes if they are properly aligned to a canonical view. MV-NRSfM is then trained to learn a decoder h d : ϕ ∈ R K → S ∈ R P ×3 which maps a low-dimensional code to an aligned 3D shape, as well as an encoder network h e : W 1 , W 2 , ..., W V → ϕ which estimates a single shape code ϕ from 2D landmarks W v ∈ R P ×2 captured from a number of different views (see Appendix C for the network architecture). Both h d and h e are learned through minimizing the reprojection error: where S refers to the training set, and • denotes function composition. Thanks to the constraint from low dimensional codes as well as the convolution structure of h e inspired from factorization-based NRSfM methods [18], the learned networks h d •h e are able to infer reasonable 3D landmark positions from noisy 2D landmark inputs. We provide the network architecture of MV-NRSfM in Fig. 10 of Appendix C.
In our task, we rely on the robustness of MV-NRSfM not only to learn the 3D reconstruction of the labeled training set, but also to detect outliers on the unlabeled set using Eq. 2. At the t-th iteration of our self-training, we train h t d , h t e given the current labeled set S t−1 from the previous iteration. We then test h t d • h t e on the detected 2D landmarks from the unlabeled set to produceS n used in Eq. 2. Camera extrinsicsT (n,v) are then estimated simply through either an orthographic-N-point (OnP) or perspective-N-point (PnP) solver depending on the choice of camera projection model. In our data, we find that assuming a weak perspective camera and use OnP already gives high fidelity results.
Finally, We note that the unsupervised learned MV-NRSfM networks i.e. h d • h e is likely not able to estimate correct 3D landmarks if its 2D inputs are significantly different than its training set. Instead, it tends to output a plausible 3D structure but does not fully match the 2D inputs. This is actually a desirable behavior for our task, since it serves the purpose of out-of-distribution detection -detects any shapes differ significantly to the current labeled set. We expect the MV-NRSfM to cover full shape variations in the input sequences as the "labeled" set expands while the training progresses. We give a detailed analysis in Fig. 8 of Appendix A.

Experiments
Our experiments aim to answer the following questions: (I) Is MBW able to generate reliable 2D and 3D landmark predictions from limited views (as few as two) given only a few (as few as 1-2%) human labels? (II): Is MBW able to reject outliers and learn a meaningful shape distribution from these few input labels? (III) How important is the number of views in our pipeline? (IV) Can MBW refine (denoise) the 2D candidate inliers? (V) Is our pipeline able to compete with leading benchmarks despite using a fraction of input 2D labels? Before diving into our experiments, we discuss the details of our pipeline.
Datasets. Datasets with multi-view videos of non-human subjects are rare, so we collect our own dataset of animals. The collected zoo dataset consists of five animal categories, each with 2 synchronized videos. The videos contain viewpoint and dynamic appearance changes as well as common imaging artifacts such as reflection of water or blurred frames (see Fig. 3). For this data we manually annotated part of the sequences for evaluation. In addition, we used the benchmark dataset of Human3.6M [12] (H36M) to perform quantitative evaluations of our approach. Implementation Details We train our approach on 1 NVIDIA RTX 3090 GPU with 24 GB memory. A learning rate of 0.001 is used for all networks. We train each network from scratch. A pre-trained RAFT network [35] is used with flow iterations of 20. Bottleneck size of 8 is used for MV-NRSfM [3] for all categories. We use HRNet [34] as the backbone 2D detector, and the same configuration is used for all object categories.
Question I: Limited amount of labels and views We use just two camera views from Directions-1 sequence of Subject #1 from H36M dataset [12]. Each camera view consists of 1383 frames per-view, amounting to 2766 frames in total. Of these, we provide hand labels for only 20 frames (10 frames per view amounting to 0.8% of the total frames) through uniform sampling. Our task is to generate 2D landmark predictions of the remaining frames of this sequence ( 99.2% unlabeled).
We evaluate the accuracy of 2D landmark predictions using the commonly used evaluation metric of PCK by Andriluka et al. [38]. We report area under the curve of PCK at different thresholds to understand the nature of 2D prediction errors over all the frames. For consistency, the 2D landmark error is normalized using head bone length before evaluation. As baseline, we keep the complete architecture of MBW, but replace MV-NRSfM with multi-view triangulation using groundtruth calibrated cameras to reject outliers and denoise inliers [33]. We denote this baseline as Trng.
2D landmark prediction performance over all the frames is shown in Fig. 4c. The quantitative results are shown in Tab. 2 where we we report PCK AUC values to evaluate 2D landmark prediction accuracy. We evaluate the 3D structure accuracy using Procrustes-Aligned Mean Per Joint Position Error (PA-MPJPE) [32]. This metric evaluates 3D joint localization accuracy in mm and represents the L2 distance between the groundtruth and predicted joint locations after aligning the 3D structures using a rigid transformation. Table 2 shows that our approach is able to generate high-fidelity 2D landmark prediction as well as accurate 3D structure despite starting from a mere 0.8% of 2-View data. In contrast, the competing baseline fails since it cannot reconstruct good 3D structure from just 2 views and extremely noisy landmark predictions. This experiment helps us answer Question (1): Yes, MBW with MV-NRSfM is able to predict reasonable 2D and 3D landmark prediction using small amount of labels and views.
(a) Outlier detection on [12]. (b) Outlier detection on Zoo dataset (c) 2D accuracy using PCKh [38]. Figure 4: (a) Precision-Recall (PR) AUC shows the outlier rejection accuracy on benchmark [12] with varying number of camera views and input 2D labels. PR AUC bars per each view are calculated using the frames available in that view. Fewer views corresponds to a smaller number of frames, and more views comprise a larger number of frames. (b) Outlier rejection accuracy using PR AUC on zoo dataset. (c) Final 2D landmark prediction accuracy using our approach compared to multi-view triangulation (with groundtruth cameras) [11] (Trng). We calculate the normalized error (dividing by head bone length [38]) and report the PCK AUC result varying the error threshold range. Note that we run this experiment on just 10 frames per view (amounting to 0.8%) and only two views. Question II: Outlier rejection The proposed pipeline requires bad 2D landmark candidates (outliers) to be rejected so they are not incorporated into subsequent training iterations as (inlier) pseudolabels S i . To evaluate this, we design the following experiment to evaluate the Precision-Recall AUC on our zoo dataset as shown in Fig. 4b. The number of initial 2D manual labels provided to each object is 2%. We perform an ablation study in which we evaluate at the initial iteration of MBW, to check MBW's ability to reject and refine noisy landmark predictions from optical flow.
MBW is compared against weak-perspective Tomasi-Kanade Structure-from-Motion (TK) [36] -the only other approach able to reconstruct 3D structure without calibrated cameras. For completeness, we also estimate cameras in the wild using [31] so that we can compare against Trng [10]. For the competing approaches, we use our whole pipeline as-is except for the MV-NRSfM outlier rejection and denoising part, instead using the competing method. Since there is no groundtruth in the zoo dataset, we manually label each 2D flow prediction and assign inlier labels by visual verification. The PR AUC is calculated using this manually generated groundtruth verification. As evident in Fig. 4b, only our approach can accurately reject the outliers as evident by high PR AUC. This finding is reasonable since we found that [31] is unable to calculate cameras due to non-existent or poor matching features across views collected in the zoo dataset, resulting in no 3D reconstruction. Another baseline, weak-perspective Tomasi Kanade SfM is unable to handle the noise encountered in real-world data and hence fails to accurately reconstruct 3D. Thus, the above experiment helps us answer Question (II): Yes, MBW is able to reject outliers despite of having limited input 2D labels, number of views, and no camera information.
Question III: Number of Views For this experiment, we take the same H36M dataset as noted in Question 1, with the exception that we vary the number of views (2-4) and number of 2D input labels (1-2%) provided to our approach and competing baseline of Trng. In Fig. 4a we see that both our approach and the baseline reject outliers reasonably well for four views. However as we reduce the number of views Trng. had a steeper dropoff due to unreliable geometric constraints arising from limited views and noisy 2D labels. In contrast, since MBW uses a learned shape prior for rejecting outliers, our performance remains consistent across varying views. This experiment helps us answer Question (III): Yes, as long as we learn a good shape prior, we do not require large number of views to reliably reject the outliers.
Question IV: Denoising capability To answer this question, we report the visual results of the 2D landmark predictions by MBW on zoo dataset in Fig. 5. On the left, we see that MBW is able to detect bad 2D predictions without human intervention (also evident by PR AUC in Fig. 4b). On the Figure 5: (a) Shows example outliers detected using the proposed approach. (b) Shows the denoising capability of the proposed approach where we are able to denoise the improve the inliers by using the multi-view shape prior information.  [13] 19.9 Reddy et al. [28] 17.5 Ours (PA-MPJPE at 2%) 20.15 right, we highlight scenarios where our approach can use the MV-NRSfM shape prior information to not only detect but refine (denoise) the inliers. Moreover, apart from qualitative visual inspection, we also conduct a toy study to analyze the denoising capabilities of our approach and discuss this in the supplementary section. The visual inspection experiment in Fig. 5 helps us answer Question (IV): Yes, MBW can effectively denoise candidate inliers.
Question V: Comparison against leading benchmarks Finally, we compare the performance of our approach with existing state-of-the-art semi-supervised learning approaches. Since our approach has overlap with MSBR [42] in that they have an initial labeled set, as well as an unlabeled set, we compare and report the 2D landmark prediction accuracy using the same experimental setup as theirs and report in Tab. 4. To reiterate, similar to them we use Eating and Discussion but just use 2% as input labeled data and test on all frames of Greeting for a fair comparison. We observe that in spite of using a small amount of 2D input labels, our approach is able to compete against well-established benchmark approaches that use >30% of frames to label -making our approach more efficient in terms of effort required to generate labels of similar accuracy.
Lastly, we conduct 3D reconstruction analysis on the Directions-1 sequence of H36M [12] using the common protocol where Subjects 1,5,6,7,8 are used as train sets, and S9, and 11 are used as the test set. We just provide 2% of train set labels as 2D input labels to our approach and evaluate the 3D reconstruction performance on 100% of the test set, and report the 3D reconstruction results in Tab. 3. Since our approach calculates 3D reconstruction in a canonical frame and is up-to-scale, we apply the rigid transformation of Procrustes Alignment and report the 3D landmark reconstruction accuracy as PA-MPJPE in mm. We observe that we are able to compete with state-of-the-art benchmarks despite using 2% input labels and no camera information in our approach. The above two experiments helps us answer Question (V).

Limitations
Our approach relies on a 3D neural shape prior to address challenging scenarios such as occluded or noisy keypoints. However, it is only able make accurate predictions if the views have sufficiently wide baseline, for instance to address cases where an occluded keypoint in one view is unoccluded in another view. In these scenarios MBW can reject (or clean) the occluded keypoint through the 3D prior. However if the keypoint is occluded in both (all) the available views, then MBW fails to produce meaningful predictions. This is expected since MBW cannot hallucinate without any cues or priors. Some more recent works such as particle trajectory tracking [9] may help address the issues of occlusion, and we plan to take these lines of work into account in future work.

Conclusion
The idea of leveraging geometry to make labeling easier is not new -indeed this is the basis for largescale dome-based data collection methods [14, 1, 24, 44, 17]. Our key contribution is incorporating a neural prior to enable the same ideas to be applied to tail-end distribution non-rigid object categories. This has yielded several key insights: (a) Deep 3D neural shape priors trained on even as few as 10 − 15 frames per video can already provide powerful constraints for modeling deformable objects; (b) provided a reliable outlier rejection method, optical flow-based methods [35] provide a simple and effective way to propagate 2D candidate predictions in videos; (c) neural priors can be used as the basis of reliable outlier rejection, enabling bootstrapping under constrained and error-prone methods such as flow and "partially trained" 2D networks into highly accurate vision pipelines. Our approach is just a first step in this direction, but it may point the way to a revival of iterative pipelines that more deeply integrate the insights of recent neural approaches with iterative refinement methods familiar in classic 3D computer vision. [

Supplementary Material A Ablation study -Iterations in MBW
In this section, we conduct an ablation study analyzing the effects of iterations in our proposed approach. In other words, we discuss the improvements in 2D and 3D landmark predictions as well as the implications of iterations in our proposed approach. We conduct our ablation study on the publicly available human benchmark dataset [12]. We initialise our pipeline (MBW) with 5% 2D input labels and 4 views from "Directions-1" sequence of "Subject-1" [12].
Improvement in 2D and 3D landmark predictions: As shown in Fig. 7a, we see that the major improvement in 2D landmark predictions could be observed between Iteration 0 and Iteration 1.
Moreover, we see that as the iterations progress, the 2D landmark prediction error continues to reduce as seen in Fig. 7a. Furthermore, we notice that as the iterations progress, our pipeline continues to further denoise and improve the 2D landmark predictions as well as continues to generate accurate pseudo-labels as visible in Fig. 7b. Similarly, we see that as the iterations progress, the 3D reconstruction error (in PA-MPJPE) continues to decrease as visible in Tab. 5.
We also graphically visualize the effects of MBW at each stage in Fig. 8. With a learned MV-NRSfM over given data, we visualize the first two dimensions of the bottleneck. The initial two dimensions of the bottleneck show the overall spread of the given data. The red dots in this plot represents the initial set of 2D input labels. We color code this scatter plot based on 2D reprojection error. Specifically, the colors in Fig. 8 represent the error calculated from Eq. (2). As the iterations progress, we observe the reprojection error to continue to decrease as better 3D structures as well as 2D landmark predictions are learned iteratively.
Handling occlusions with geometry: Analyzing further, we investigate the type of improvement over different iterations. We notice that the main benefit of using learnable geometric self-supervision (see Sec. 3.2) is its capability to handle occlusions. Figure 6 shows that MBW, in conjunction with MV-NRSfM is able to denoise the 2D landmark predictions as seen in the Iter. 2 columns. Compared to Iter. 1, we observe that MV-NRSfM was able to denoise and then feed the pseudo-label to our iterative pipeline which resulted in correct annotations for cases with severe occlusions. Owing to the above observations, we show improvement of 2D landmark predictions over iterations, specifically in cases where the landmarks are occluded. Quantitative improvement is shown in Fig. 7a while the qualitative improvement is shown in Fig. 6 that shows improvements during Iter. 2, and Fig. 7b that shows improvement during Iter. 3 -where we observe the benefit of using the multi-view constraint of MV-NRSfM.
Denoising and its limitations: Since MV-NRSfM leverages the redundancy in shape variations among different frames, it is less sensitive to the variations of input views, and more capable of detecting outliers and denoising inlier 2D landmark estimates. More specifically, it has the capability of denoising the 2D inputs and providing a 3D structure based on its learned distribution. For the cases shown in Fig. 6 and Fig. 7b, we showcase the denoising capabilities of MV-NRSfM. However, we should note that MV-NRSfM is only able to denoise and refine the inlier estimates if the amount of noise in 2D input labels is small enough. There are two reasons: (i) Inaccurate camera matrix: If the 2D input is extremely noisy in one of the views, even if MV-NRSfM would degenerate to an accurate 3D structure, it would not be able to reliable project the generated 3D structure over the extremely noisy view because of inaccurate camera matrix calculated from OnP or PnP. (ii) Inaccurate 3D structure: If most of the views are noisy or if the baseline between cameras is not wide enough, MV-NRSfM cannot learn to enforce multi-view shape consistency thereby generating an inaccurate 3D structure.

B Initial input labels and Active learning
For the inital set of 2D input labels, we sample uniformly across time and views (unlike Pereira et al. [27] that uses PCA to decide which frames to label). Although we pick labels from each view initially, we carry this action in the initial iteration to learn a good 3D shape prior that enforces multi-view consistency. For the subsequent iterations, we do not necessarily require to pseudo labels for all the views of an instance. Figure 8: We plot the first two dimensions of the MV-NRSfM bottleneck since it shows the overall coverage and distribution of the sequential data. The red points represent the frames that were given initial 2D input labels. The colorbar of these scatter plots represents the reprojection error (Eq. (2)) between the projected 3D structure from MV-NRSfM and 2D candidate predictions at the corresponding iteration. We observe that during the final iteration, the error reduces substantially as visible by colorbars indicating that the MV-NRSfM network generalizes reasonably well with limited initial labels, and during iterations, it expands its coverage over most of the samples in the dataset.
In the experiments shown in Sec. 4, we did not have a second set of manual annotations (active learning). We observe that since we propose a principled way to detect outliers, our pipeline could be readily used in the active learning domain where our proposed approach of iterations can be useful to dictate the next set of labeling in active learning. Thus, if we have a contiguous chunk of outliers in space and time for our captured video sequence, MBW could find this chunk of outliers in a principled way and is able to exactly specify where we should sample and collect more data for the next iteration if it is used for active learning.
From the above discussion, it is clear that our approach has a rich connection with active learning and could be readily used with the work from Feng et al. [6] that uses active learning to iteratively improve the performance of the network in each iteration. Although it is outside of the scope of the paper, we would like to note that the strengths of our results show its applicability in the active learning scenario.

C MV-NRSfM Architecture Details
As shown in Fig. 9, a 3D structure is drawn from a statistical shape distribution, and consequently projected to 2 or more views using a Perspective-n-Point or Orthographic-n-Point solver. Given a set of frames, the parameters of the shape distribution are adapted by minimizing the error between the predicted and the ground truth 2D keypoints.
The architecture of the multi-view neural shape prior is shown in Fig. 10. Motivated by hierarchical sparse coding [18], we implement the neural shape prior with an autoencoder with a bottleneck dimension of 8 (we keep the same bottleneck dimension for all of our experiments across all different object categories). First, the network h e extracts the block sparse codes, Ψ. Thereafter, the bottleneck layer factorizes the block sparse code into a projection (camera) matrix and the unrotated vector sparse code, ϕ. We use the same encoder over all additional views. The vector sparse codes at the bottleneck stage are then pooled together and fed into a shape decoder, h d to generate the 3D shape at the canonical pose. Finally, the canonical 3D structure is projected over all the views, using the closed-form solution of the PnP or OnP solver.

D Bounding Box calculation
We assume that only a single object of interest (Chimpanzee in Fig. 11 is visible in each frame. In this section, we briefly explain our technique to estimate bounding boxes to reduce the problem into a single object. Using the initial set of 2D input labels, we propagate optical flow [35] and generate Figure 9: The 3D structure is drawn from a statistical shape distribution using neural shape priors and consequently projected to multiple views using the cameras calculated through OnP formulation. MV-NRSfM minimizes the 2D reprojection error between the predicted 2D projections and target (input) 2D projections. Figure 10: Architecture showing autoencoder-based MV-NRSfM approach. The 2D inputs (targets) from multiple views act as an input to encoder h e that extracts the block sparse code Ψ from the corresponding views. A Rotation Factorization (RF) layer at the bottleneck stage, factorizes the block sparse code into the respective camera matrices and the unrotated vector sparse code ϕ. The codes are then fused via pooling function into a single code that acts as an input to the shape decoder h d . The shape decoder predicts the 3D structure in the canonical frame while enforcing equivariant view consistency.
2D candidate predictions over the entire sequence. During this initial (flow iteration) iteration, we calculate the bounding box for the entire sequence using the 2D candidate predictions -by taking the smallest and largest x and y coordinates of 2D candidate predictions. However, since the 2D candidate predictions coming from the optical flow are unreliable and noisy, we pad the bounding box with extra space by a fixed size to make the object visible, as shown in Fig. 11a. For the subsequent iterations, we calculate the bounding boxes from the 2D detector network predictions that are denoised through MV-NRSfM. Since we already denoise the 2D candidate predictions, we remove the extra padding from the bounding box calculation. An experimental video showing the bounding box visualization improvement from initial to final iteration is attached in the supplementary.

E MBW-Zoo Dataset
The URL to platform where the dataset can be viewed and downloaded: https://github.com/mosamdabhi/MBW-Data. Further information concerning the (a) Bounding box during initial iteration.
(b) Bounding box during subsequent iterations. Figure 11: (a) Bounding box visualization during the initial iteration -calculated from the optical flow 2D landmark predictions. Unless readjusted, since optical flow predictions tend to accrue errors, we see that the calculated bounding box exhibit a similar error (elongation to the right). (b) During subsequent iterations, the bounding box predictions become accurate due to less noisy 2D candidate predictions.
released data asset such as datasheets for dataset, and overall dataset documentation is discussed in the following subsections.

E.1 Overview
We release a challenging dataset and consisting image frames of tail-end distribution categories (such as Fish, Colobus Monkeys, Chimpanzees, etc.) with their corresponding 2D, 3D, and Bounding-Box labels generated from minimal human intervention. Some of the prominent use cases of this dataset include not only sparse 2D and 3D landmark prediction, but also dense reconstruction tasks such as dense deformable shape reconstruction, novel view rendering (NeRF), and finally this dataset could also be used for advancing Simultaneous Localization and Mapping (SLAM) frameworks. We also submit the codebase that we used to generate the above labels for in-the-wild object categories.
The data was collected by two smartphone cameras without any constraints: meaning no guidance or instructions were given as to how the data should be collected. The intention was to mimic the data captured casually by anyone holding a smartphone grade camera. Due to this reason, the cameras were continuously moving in space changing their extrinsics with respect to each other, capturing an in-the-wild dynamic scene. Thus, this dataset could be used to benchmark robust algorithms in various computer vision tasks.

E.2 Collection process
We capture 2-View videos from handheld smartphone cameras. In our case, we used an iPhone 11 Pro Max and an iPhone 12 Pro Max to capture the video sequences. We use Final Cut Pro to manually synchronize the 2-View video sequences using the audio signal and time stamps. Please note that all we require are 2-view synchronized image frames and manual annotations for 1-2% of the data. No camera calibration (intrinsics or extrinsics) is required to run MBW. Figure 12: Overview of the dataset. The dataset consists of 2-view synchronized video sequences captured from a zoo visit. MBW provides landmark predictions (labels) that we provide for the categories shown above.

E.3 Frames visualization
In total, there are 16154 instances in this dataset from 7 different object categories, coming from 2 camera views. Sample instances are visualized in Fig. 13. Figure 13: Visualization of instances from 7 different object categories.

E.4 Joint connections visualization
We manually annotate 1-2% of the image frames per view. Our annotation consists of the 2D landmark keypoints. The location of landmarks is chosen to extract articulated information from the objects. Joint connection visualization is shown below.
How to choose the number of joints to track? Different applications require the tracking of different keypoints (landmarks/joints) of tail-end distribution objects. Thus, our approach (MBW) gives user the freedom to decide which keypoints are of interest and hence should be chosen to track.
In the released dataset, we chose keypoints that explain the articulation or deformation of the object catgeory, that we visualize the in Fig. 14.

F Dataset format
The dataset (can be viewed and downloaded by the reviewers from here [2]. The downloaded dataset is divided into two directories: annot/ and images/. As names suggest, the annot/ directory contains annotations and images/ directory consists of 2-view synchronized image frames.

Annotations format
The annotation format is discussed in Tab. 6. Bounding box crops generated from MBW confidence Flag specifying confidence (Eq. (2)) for the MBW predictions.

G Datasheets for dataset
As part of making dataset collection more easy and amenable in a wildly unconstrained setup, we collect a dataset of zoo animals using smartphone grade cameras, and annotate (≈ 2%) of the collected frames with 2D keypoint landmarks. We call this dataset the Multiview Bootstrapping in the wild (MBW) Zoo dataset; what follows below is the datasheet [7] describing this data.

1.
For what purpose was the dataset created? (Was there a specific task in mind? Was there a specific gap that needed to be filled? Please provide a description.) This dataset was created to generate 2D and 3D labels of articulated objects in unconstrained settings. Such unconstrained and casually collected datasets have a wide variety of applications including entertainment, neuroscience, psychology, ethology, and many fields of medicine. Large offline labeled datasets do not exist for all but the most common articulated object categories (e.g., humans, hands, cars). Hand labeling these landmarks within a video sequence is a laborious task. Learned landmark detectors can help, but can be error-prone when trained from only a few examples. As part of contribution of this paper, we provide this dataset where 2D and 3D labels are generated in very challenging scenarios, using our approach. Note that the user is required to only provide handheld videos from 2 or more views and manual 2D keypoint labels for 15 frames per video. No camera intrinsics or extrinsics information is required to generate the labels shown in this dataset.
2. Who created this dataset? (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)? Each instance is an image of an articulated object (zoo animals, birds, and fish). Approximately 2-5% of images have corresponding manual 2D landmark annotation. From here on, we use the term landmark prediction and label interchangeably for convenience. For the remaining images in the sequences, the 2D and 3D landmark labels, as well as bounding box crops labels are generated by MBW. In particular, each entity also has a confidence flag that is a boolean specifying how much reliable is the label generated by MBW.
2. How many instances are there in total (of each type, if appropriate)?
In total, there are 16154 instances in this dataset from 7 different object categories, coming from 2 camera views. The overall dataset statistics in Tab. 7 reflect the above description. We are unable to provide the stereo baseline distance (m) and stereo angle ( • ) since the data was captured where the cameras were continuously moving thereby changing these metrics.
describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable)?
It is a sample of all videos captured casually in an unconstrained environment such as zoo. It is not intended to be representative: the data was collected randomly in the order of visit. This data was collected with the intent to show the applicability of MBW in challenging data scenarios -specifically to label articulated objects in the wild at scale. 4. What data does each instance consist of? ("Raw" data (e.g., unprocessed text or images)or features? In either case, please provide a description).
Please refer Sec. F and the corresponding Table 6. 5. Is there a label or target associated with each instance? If so, please provide a description.
As noted in the Table 6, labels for each instance are associated for the categories with flag Available. For the rest, only initial ≈ 2% W_GT labels are provided, since we did not run MBW that could provide us with prediction labels. We release this dataset to set a benchmark for solving challenging 2D and 3D landmark prediction tasks for in-the-wild unconstrained video captures. 6. Is any information missing from individual instances? (If so, please provide a description, explaining why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text.)* The prediction labels (W_Predictions , S_Pred , BBox, and confidence ) are missing for the categories where MBW is not run as shown in Table 6. 7. Are relationships between individual instances made explicit? (e.g., users' movie ratings, social network links)? If so, please describe how these relationships are made explicit.
Instances are unrelated.
8. Are there recommended data splits (e.g., training, development/validation, testing)? If so, please provide a description of these splits, explaining the rationale behind them.
Since the sole purpose of this data collection was to generate labels from scratch, we expect this data to be used solely for generating labels for unlabeled data. Thus, we do not explicitly provide a training/validation/testing split; however, we recognize that people may wish to do this, or to do some form of cross-validation. We would suggest cross-validation and test split by dividing the manual labels into 80/10/10 split and pick the samples via a uniform sampling strategy. 9. Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide a description.
Since the inital 2D keypoints were manually labeled, there could be some errors in the manual annotations since they were visually localized and clicked as labels. 10. Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? If it links to or relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) are there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to a future user? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, as appropriate.) The dataset needs to be downloaded from here [2].