@inproceedings{
pan2024actanywhere,
title={ActAnywhere: Subject-Aware Video Background Generation},
author={Boxiao Pan and Zhan Xu and Chun-Hao Paul Huang and Krishna Kumar Singh and Yang Zhou and Leonidas Guibas and Jimei Yang},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=ntlFREw59A}
}
We study a novel problem to automatically generate video background that tailors to foreground subject motion. It is an important problem for the movie industry and visual effects community, which traditionally requires tedious manual efforts to solve. To this end, we propose ActAnywhere, a video diffusion model that takes as input a sequence of foreground subject segmentation and an image of a novel background and generates a video of the subject interacting in this background. We train our model on a large-scale dataset of 2.4M videos of human-scene interactions. Through extensive evaluation, we show that our model produces videos with realistic foreground-background interaction while strictly following the guidance of the condition image. Our model generalizes to diverse scenarios including non-human subjects, gaming and animation clips, as well as videos with multiple moving subjects. Both quantitative and qualitative comparisons demonstrate that our model significantly outperforms existing methods, which fail to accomplish the studied task. Please visit our project webpage at https://actanywhere.github.io.
ActAnywhere: Subject-Aware Video Background Generation
Boxiao Pan Zhan Xu Chun-Hao Paul Huang Krishna Kumar Singh Yang Zhou Leonidas Guibas Jimei Yang