@inproceedings{
zhang2024from,
title={From Speaker to Dubber: Movie Dubbing with Prosody and Duration Consistency Learning},
author={Zhedong Zhang and Liang Li and Gaoxiang Cong and Haibing YIN and Yuhan Gao and Chenggang Yan and Anton van den Hengel and Yuankai Qi},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=QHRNR64J1m}
}
Movie Dubbing aims to convert scripts into speeches that align with the given movie clip in both temporal and emotional aspects while preserving the vocal timbre of one brief reference audio.
The wide variations in emotion, pace, and environment that dubbed speech must exhibit to achieve real alignment make dubbing a complex task.
Considering the limited scale of the movie dubbing datasets (due to copyright) and the interference from background noise, directly learning from movie dubbing datasets limits the pronunciation quality of learned models.
To address this problem, we propose a two-stage dubbing method that allows the model to first learn pronunciation knowledge before practicing it in movie dubbing.
In the first stage, we introduce a multi-task approach to pre-train a phoneme encoder on a large-scale text-speech corpus for learning clear and natural phoneme pronunciations.
For the second stage, we devise a prosody consistency learning module to bridge the emotional expression with the phoneme-level dubbing prosody attributes (pitch and energy).
Finally, we design a duration consistency reasoning module to align the dubbing duration with the lip movement.
Extensive experiments demonstrate that our method outperforms several state-of-the-art methods on two primary benchmarks.
The source code and model checkpoints will be released to the public.
The demos are available at https://speaker2dubber.github.io/.
From Speaker to Dubber: Movie Dubbing with Prosody and Duration Consistency Learning
Zhedong Zhang Liang Li Gaoxiang Cong Haibing YIN Yuhan Gao Chenggang Yan Anton van den Hengel Yuankai Qi