@inproceedings{
wang2024tiva,
title={Ti{VA}: Time-Aligned Video-to-Audio Generation},
author={Xihua Wang and Yuyue Wang and Yihan Wu and Ruihua Song and Xu Tan and Zehua Chen and Hongteng Xu and Guodong Sui},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=tIpOYtxerl}
}
Video-to-audio generation is crucial for autonomous video editing and post-processing, which aims to generate high-quality audio for silent videos with semantic similarity and temporal synchronization. However, most existing methods mainly focus on matching the semantics of the visual and acoustic modalities while merely considering their temporal alignment in a coarse granularity, thus failing to achieve precise synchronization. In this study, we propose a novel time-aligned video-to-audio framework, called TiVA, to achieve semantic matching and temporal synchronization jointly when generating audio. Given a silent video, our method encodes its visual semantics and predicts an audio layout separately. Then, leveraging the semantic latent embeddings and the predicted audio layout as condition, it learns a latent diffusion-based audio generator. Comprehensive objective and subjective experiments demonstrate that our method consistently outperforms state-of-the-art methods on semantic matching and temporal synchronization.
TiVA: Time-Aligned Video-to-Audio Generation
Xihua Wang Yuyue Wang Yihan Wu Ruihua Song Xu Tan Zehua Chen Hongteng Xu Guodong Sui