To leverage crowd-sourced data to train multi-speaker text-to-speech (TTS) models that can synthesize clean speech for all speakers, it is essential to learn disentangled representations which can independently control the speaker identity and background noise in generated signals. However, learning such representations can be challenging, due to the lack of labels describing the recording conditions of each training example, and the fact that  speakers and recording conditions are often correlated, e.g. since users often make many recordings using the same equipment. This paper proposes three components to address this problem by: (1) formulating a conditional generative model with factorized latent variables, (2) using data augmentation to add noise that is not correlated with speaker identity and whose label is known during training, and (3) using adversarial factorization to improve disentanglement. Experimental results demonstrate that the proposed method can disentangle speaker and noise attributes even if they are correlated in the training data, and can be used to consistently synthesize clean speech for all speakers. Ablation studies verify the importance of each proposed component. Recent development of neural end-to-end TTS models BID26 BID1 enables control of both labelled and unlabelled speech attributes by conditioning synthesis on both text and learned attribute representations BID27 BID21 BID10 BID0 BID5 BID9 . This opens the door to leveraging crowd-sourced speech recorded under various acoustic conditions BID18 to train a high-quality multi-speaker TTS model that is capable of consistently producing clean speech. To achieve this, it is essential to learn disentangled representations that control speaker and acoustic conditions independently. However, this can be challenging for two reasons. First, the underlying acoustic conditions of an utterance, such as the type and level of background noise and reverberation, are difficult to annotate, and therefore such labels are often unavailable. This hinders the use of direct conditioning on the acoustic condition labels in a way similar to conditioning on one-hot speaker labels BID1 . Second, speaker identity can have strong correlations with recording conditions, since a speaker might make most of their recordings in the same location using the same device. This makes it difficult to learn a disentangled representation by assuming statistical independence BID6 .We address this scenario by introducing three components: a conditional generative model with factorized latent variables to control different attributes, data augmentation by adding background noise to training utterances in order to counteract the inherent speaker-noise correlation and to create ground truth noisy acoustic condition labels, and adversarial training based on the generated labels to encourage disentanglement between latent variables. We utilize the VCTK speech synthesis dataset BID23 , and background noise signals from the CHiME-4 challenge BID24 to synthesize a dataset containing correlated speaker and background noise conditions for controlled experiments. We extensively evaluate disentanglement performance on the learned latent representations as well as the synthesized samples. Experimental results identify the contribution of each component, and demonstrate the ability of the proposed model to disentangle noise from speakers and consistently synthesize clean speech for all speakers, despite the strong correlation in the training data. We build a neural network TTS model which incorporates conditional generative modeling, data augmentation, and adversarial training to learn disentangled representations of correlated and partially unlabeled attributes, which can be used to independently control different aspects of the synthesized speech. Extensive studies on a synthetic dataset verify the effectiveness of each element of the proposed solution, and demonstrate the robustness to the choice of hyperparameters.The proposed methods for disentangling correlated attributes is general, and can potentially be applied to other pairs of correlated factors, such as reverberation and speaker, or to other modalities, such as controllable text-to-image generation. In addition, for future work, we would also like to investigate the capability of the proposed method to disentangle pairs of attributes which are both unsupervised.6 Acknowledgement
Data augmentation and adversarial training are very effective for disentangling correlated speaker and noise, enabling independent control of each attribute for text-to-speech synthesis.
{"src": "scitldr-AIC_train", "paper_id": "Bkg9ZeBB37"}