Learning-based 4D reconstruction.
Reconstruction of the complete shape of dynamic objects is considered hard due to big intra-class variations.
For popular object categories such as humans or certain animals, supervised learning of object template parameters has been utilized to ease reconstruction [1]}, [2]}, [3]}, [4]}, [5]}. These templates are carefully designed parametric 3D models, which restrict learning to a low-dimensional solution space. However, the expressiveness of templates is limited. For instance, the SMPL [3]} human template struggles to capture the clothing or hair styles. In addition, systems relying on templates are class-specific as different object categories need to be parameterized differently. We think it remains illusive to construct a template for every object category.
To overcome these issues, OFlow [7]} directly learns 4D reconstruction from scratch. However, it predicts articulation in normalized model space and thus overlooks rigid-motion of the object. It also struggles to handle occlusion.
When ground-truth 3D models are not available, dynamic neural radiance fields (NeRF) [8]}, [9]}, [10]}, [11]} learn an implicit scene representation from videos, but are often scene-specific and don't scale to a class-agnostic setting. Self-supervised methods [12]}, [13]}, [14]} are promising and learn 4D reconstruction via 2D supervision of differentiable rendering [15]}, [16]}.
In contrast, in this work, we present a class-agnostic and template-free framework which learns to recover the shape and dynamics from input videos.
<FIGURE>