@inproceedings{
xiao2024eggesture,
title={{EGG}esture: Entropy-Guided Vector Quantized Variational AutoEncoder for Co-Speech Gesture Generation},
author={yiyong xiao and Kai Shu and Haoyi Zhang and BaoHuaYin and Wai Seng Cheang and Haoyang Wang and Jiechao Gao},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=Xmapdbf1CT}
}
Co-Speech gesture generation encounters challenges with imbalanced, long-tailed gesture distributions. While recent methods typically address this by employing Vector Quantized Variational Autoencoder (VQ-VAE), encode gestures into a codebook and classify codebook indices based on audio or text cues. However, due to the imbalanced, the codebook classification tends to bias towards majority gestures, neglecting semantically rich minority gestures. To address this, this paper proposes the Entropy-Guided Co-Speech Gesture Generation (EGGesture). EGGesture leverages an Entropy-Guided VQ-VAE to jointly optimize the distribution of codebook indices and adjust loss weights for codebook index classification, which consists of a) A differentiable approach for entropy computation using Gumbel-Softmax and cosine similarity, facilitating online codebook distribution optimization, and b) a strategy that utilizes computed codebook entropy to collaboratively guide the classification loss weighting. These designs enable the dynamic refinement of the codebook utilization, striking a balance between the quality of the learned gesture representation and the accuracy of the classification phase. Experiments on the Trinity and BEAT datasets demonstrate EGGesture’s state-of-the-art performance both qualitatively and quantitatively. The code and video are available.
EGGesture: Entropy-Guided Vector Quantized Variational AutoEncoder for Co-Speech Gesture Generation
yiyong xiao Kai Shu Haoyi Zhang BaoHuaYin Wai Seng Cheang Haoyang Wang Jiechao Gao