@inproceedings{
liu2024heromaker,
title={HeroMaker: Human-centric Video Editing with Motion Priors},
author={Shiyu Liu and Zibo Zhao and Yihao Zhi and Yiqun Zhao and Binbin Huang and Shuo Wang and Ruoyu Wang and Michael Xuan and Zhengxin Li and Shenghua Gao},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=jUeqWFsK1p}
}
Video generation and editing, particularly human-centric video editing, has seen a surge of interest in its potential to create immersive and dynamic content. A fundamental challenge is ensuring temporal coherence and visual harmony across frames, especially in handling large-scale human motion and maintaining consistency over long sequences. The previous methods, such as diffusion-based video editing, struggle with flickering and length limitations. In contrast, methods employing Video-2D representations grapple with accurately capturing complex structural relationships in large-scale human motion. Simultaneously, some patterns on the human body appear intermittently throughout the video, posing a knotty problem in identifying visual correspondence. To address the above problems, we present HeroMaker. This human-centric video editing framework manipulates the person's appearance within the input video and achieves inter-frame consistent results. Specifically, we propose to learn the motion priors, transformations from dual canonical fields to each video frame, by leveraging the body mesh-based human motion warping and neural deformation-based margin refinement in the video reconstruction framework to ensure the semantic correctness of canonical fields. HeroMaker performs human-centric video editing by manipulating the dual canonical fields and combining them with motion priors to synthesize temporally coherent and visually plausible results. Comprehensive experiments demonstrate that our approach surpasses existing methods regarding temporal consistency, visual quality, and semantic coherence.
HeroMaker: Human-centric Video Editing with Motion Priors
Shiyu Liu Zibo Zhao Yihao Zhi Yiqun Zhao Binbin Huang Shuo Wang Ruoyu Wang Michael Xuan Zhengxin Li Shenghua Gao