@inproceedings{
dong2024realistic,
title={Realistic Full-Body Motion Generation from Sparse Tracking with State Space Model},
author={Kun Dong and Jian Xue and Zehai Niu and Xing Lan and Ke Lv and Qingyuan Liu and Xiaoyu Qin},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=dA6yat7UNM}
}
In the domain of generative multimedia and interactive experiences, generating realistic and accurate full-body poses from sparse tracking is crucial for many real-world applications, while achieving sequence modeling and efficient motion generation remains challenging. Recently, state space models (SSMs) with efficient hardware-aware designs (i.e., Mamba) have shown great potential for sequence modeling, particularly in temporal contexts. However, processing motion data is still challenging for SSMs. Specifically, the sparsity of input conditions makes motion generation an ill-posed problem. Moreover, the complex structure of the human body further complicates this task. To address these issues, we present Motion Mamba Diffusion (MMD), a novel conditional diffusion model, which effectively utilizes the sequence modeling capability of SSMs and the robust generation ability of diffusion models to track full-body poses accurately. In particular, we design a bidirectional Temporal Mamba Module (TMM) to model motion sequence. Additionally, a Spatial Mamba Module (SMM) is further proposed for feature enhancement within a single frame. Extensive experiments on the large motion capture dataset (AMASS) demonstrate that our proposed approach outperforms the latest methods in terms of accuracy and smoothness and achieves new state-of-the-art performance. Moreover, our approach runs in real-time, making it ideal for employment in practical applications. The source code will be made public upon acceptance of this paper.
Realistic Full-Body Motion Generation from Sparse Tracking with State Space Model
Kun Dong Jian Xue Zehai Niu Xing Lan Ke Lv Qingyuan Liu Xiaoyu Qin