@inproceedings{
chi2024estimating,
title={Estimating Ego-Body Pose from Doubly Sparse Egocentric Video Data},
author={Seunggeun Chi and Pin-Hao Huang and Enna Sachdeva and Hengbo Ma and Karthik Ramani and Kwonjoon Lee},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=MHCnLo2QeA}
}
We study the problem of estimating the body movements of a camera wearer from egocentric videos. Current methods for ego-body pose estimation rely on temporally dense sensor data, such as IMU measurements from spatially sparse body parts like the head and hands. However, we propose that even temporally sparse observations, such as hand poses captured intermittently from egocentric videos during natural or periodic hand movements, can effectively constrain overall body motion. Naively applying diffusion models to generate full-body pose from head pose and sparse hand pose leads to suboptimal results. To overcome this, we develop a two-stage approach that decomposes the problem into temporal completion and spatial completion. First, our method employs masked autoencoders to impute hand trajectories by leveraging the spatiotemporal correlations between the head pose sequence and intermittent hand poses, providing uncertainty estimates. Subsequently, we employ conditional diffusion models to generate plausible full-body motions based on these temporally dense trajectories of the head and hands, guided by the uncertainty estimates from the imputation. The effectiveness of our methods was rigorously tested and validated through comprehensive experiments conducted on various HMD setup with AMASS and Ego-Exo4D datasets.
Estimating Ego-Body Pose from Doubly Sparse Egocentric Video Data
Seunggeun Chi Pin-Hao Huang Enna Sachdeva Hengbo Ma Karthik Ramani Kwonjoon Lee