@inproceedings{
lin2024action,
title={Action Imitation in Common Action Space for Customized Action Image Synthesis},
author={Wang Lin and Jingyuan Chen and Jiaxin Shi and Zirun Guo and Yichen Zhu and Zehan Wang and Tao Jin and Zhou Zhao and Fei Wu and Shuicheng YAN and Hanwang Zhang},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=h2e4G2YiwR}
}
We propose a novel method, \textbf{TwinAct}, to tackle the challenge of decoupling actions and actors in order to customize the text-guided diffusion models (TGDMs) for few-shot action image generation. TwinAct addresses the limitations of existing methods that struggle to decouple actions from other semantics (e.g., the actor's appearance) due to the lack of an effective inductive bias with few exemplar images. Our approach introduces a common action space, which is a textual embedding space focused solely on actions, enabling precise customization without actor-related details. Specifically, TwinAct involves three key steps: 1) Building common action space based on a set of representative action phrases; 2) Imitating the customized action within the action space; and 3) Generating highly adaptable customized action images in diverse contexts with action similarity loss. To comprehensively evaluate TwinAct, we construct a novel benchmark, which provides sample images with various forms of actions. Extensive experiments demonstrate TwinAct's superiority in generating accurate, context-independent customized actions while maintaining the identity consistency of different subjects, including animals, humans, and even customized actors.
Action Imitation in Common Action Space for Customized Action Image Synthesis
Wang Lin Jingyuan Chen Jiaxin Shi Zirun Guo Yichen Zhu Zehan Wang Tao Jin Zhou Zhao Fei Wu Shuicheng YAN Hanwang Zhang