@inproceedings{
dai2024expressivesinger,
title={ExpressiveSinger: Multilingual and Multi-Style Score-based Singing Voice Synthesis with Expressive Performance Control},
author={Shuqi Dai and Ming-Yu Liu and Rafael Valle and Siddharth Gururani},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=y9J0PNOOrY}
}
Singing Voice Synthesis (SVS) has significantly advanced with deep generative models, achieving high audio quality but still struggling with musicality, mainly due to the lack of performance control over timing, dynamics, and pitch, which are essential for music expression. Additionally, integrating data and supporting diverse languages and styles in SVS remain challenging. To tackle these issues, this paper presents \textit{ExpressiveSinger}, an SVS framework that leverages a cascade of diffusion models to generate realistic singing across multiple languages, styles, and techniques from scores and lyrics. Our approach begins with consolidating, cleaning, annotating, and processing public singing datasets, developing a multilingual phoneme set, and incorporating different musical styles and techniques. We then design methods for generating expressive performance control signals including phoneme timing, F0 curves, and amplitude envelopes, which enhance musicality and model consistency, introduce more controllability, and reduce data requirements. Finally, we generate mel-spectrograms and audio from performance control signals with style guidance and singer timbre embedding. Our models also enable trained singers to sing in new languages and styles. A listening test reveals both high musicality and audio quality of our generated singing compared with existing works and human singing. We release the data for future research. Demo: We release the data for future research. Demo: https://expressivesinger.github.io/ExpressiveSinger.
ExpressiveSinger: Multilingual and Multi-Style Score-based Singing Voice Synthesis with Expressive Performance Control
Shuqi Dai Ming-Yu Liu Rafael Valle Siddharth Gururani