The modeling of style when synthesizing natural human speech from text has been the focus of significant attention. Some state-of-the-art approaches train an encoder-decoder network on paired text and audio samples (x_txt, x_aud) by encouraging its output to reconstruct x_aud. The synthesized audio waveform is expected to contain the verbal content of x_txt and the auditory style of x_aud. Unfortunately, modeling style in TTS is somewhat under-determined and training models with a reconstruction loss alone is insufficient to disentangle content and style from other factors of variation. In this work, we introduce an end-to-end TTS model that offers enhanced content-style disentanglement ability and controllability. We achieve this by combining a pairwise training procedure, an adversarial game, and a collaborative game into one training scheme. The adversarial game concentrates the true data distribution, and the collaborative game minimizes the distance between real samples and generated samples in both the original space and the latent space. As a result, the proposed model delivers a highly controllable generator, and a disentangled representation. Benefiting from the separate modeling of style and content, our model can generate human fidelity speech that satisfies the desired style conditions. Our model achieves start-of-the-art results across multiple tasks, including style transfer (content and style swapping), emotion modeling, and identity transfer (fitting a new speaker's voice). In the past few years, we have seen exciting developments in Text-To-Speech (TTS) using deep neural networks that learn to synthesize human-like speech from text in an end-to-end fashion. Ideally, synthesized speech should convey the given text content in an appropriate auditory style which we refer to as style modeling. Modeling style is of particular importance for many practical applications such as intelligent conversational agents and assistants. Yet, this is an incredibly challenging task because the same text can map to different speaking styles, making the problem somewhat under-determined. To this end, the recently proposed Tacotron-based approaches BID22 ) use a piece of reference speech audio to specify the expected style. Given a pair of text and audio input, they assume two independent latent variables: c that encodes content from text, and s that encodes style from the reference audio, where c and s are produced by a text encoder and a style encoder, respectively. A new audio waveform can be consequently generated by a decoder conditioned on c and s, i.e. p(x|c, s). Thus, it is straightforward to train the model that minimizes the log-likelihood by a reconstruction loss. However, this method makes it challenging for s to exclusively encode style because no constraints are placed on the disentanglement of style from content within the reference audio. It makes the model easy to simply memorize all the information (i.e. both style and content components) from the paired audio sample. In this case, the style embedding tends to be neglected by the decoder, and the style encoder cannot be optimized easily.To help address some of the limitations of the prior work, we propose a model that provides enhanced controllability and disentanglement ability. Rather than only training on a single paired text-audio sample (the text and audio are aligned with each other), i.e. (x txt , x aud ) →x, we adopt a pairwise training procedure to enforce our model to correctly map input text to two different audio references (x txt , x aud is paired with x txt , and x − aud is unpaired (randomly sampled). Training the model involves solving an adversarial game and a collaborative game. The adversarial game concentrates the true joint data distribution p(x, c) by using a conditional GAN loss. The collaborative game is built to minimize the distance of generated samples from the real samples in both original space and latent space. Specifically, we introduce two additional losses, the reconstruction loss and the style loss. The style loss is produced by drawing inspiration from image style transfer BID4 , which can be used to give explicit style constraints. During training, the the generator and discriminator combat each other to match a joint distribution. While at the same time, they also collaborate with each other in order to minimize the distance of the expected sample and the synthesized sample in both original space and hidden space. As a result, our model delivers a highly controllable generator and disentangled representation. We propose an end-to-end conditional generative model for TTS style modeling. The proposed model is built upon Tacotron, with an enhanced content-style disentanglement ability and controllability. The proposed pairwise training approach that involves a adversarial game and a collaborative game together, result in a highly controllable generator with disentangled representations. Benefiting from the separate modeling of content c and style s, our model can synthesize high fidelity speech signals with the correct content and realistic style, resulting in natural human-like speech. We demonstrated our approach on two TTS datasets with different auditory styles (emotion and speaker identity), and show that our approach establishes state-of-the-art quantitative and qualitative performance on a variety of tasks. For future research, an important direction can be training on unpaired data under an unsupervised setting. In this way, the requirements for a lot of work on aligning text and audios can be much released.
a generative adversarial network for style modeling in a text-to-speech system
{"src": "scitldr-AIC_train", "paper_id": "ByzcS3AcYX"}