@inproceedings{
liu2024emphasizing,
title={Emphasizing Semantic Consistency of Salient Posture for Speech-Driven Gesture Generation},
author={Fengqi Liu and Hexiang Wang and Jingyu Gong and Ran Yi and Qianyu Zhou and Xuequan Lu and Jiangbo Lu and Lizhuang Ma},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=NaUbjH7QEL}
}
Speech-driven gesture generation aims at synthesizing a gesture sequence synchronized with the input speech signal. Previous methods leverage neural networks to directly map a compact audio representation to the gesture sequence, ignoring the semantic association of different modalities and failing to deal with salient gestures. In this paper, we propose a novel speech-driven gesture generation method by emphasizing the semantic consistency of salient posture. Specifically, we first learn a joint manifold space for the individual representation of audio and body pose to exploit the inherent semantic association between the two modalities, and propose to enforce semantic consistency via a consistency loss. Furthermore, we emphasize the semantic consistency of salient postures by introducing a weakly-supervised detector to identify salient postures, and reweighting the consistency loss to focus more on learning the correspondence between salient postures and the high-level semantics of speech content. In addition, we propose to extract audio features dedicated to facial expression and body gesture separately, and design separate branches for face and body gesture synthesis. Extensive experiments and visualization results demonstrate the superiority of our method over the state-of-the-art approaches.
Emphasizing Semantic Consistency of Salient Posture for Speech-Driven Gesture Generation
Fengqi Liu Hexiang Wang Jingyu Gong Ran Yi Qianyu Zhou Xuequan Lu Jiangbo Lu Lizhuang Ma