Voice Conversion (VC) is a task of converting perceived speaker identity from a source speaker to a particular target speaker. Earlier approaches in the literature primarily find a mapping between the given source-target speaker-pairs. Developing mapping techniques for many-to-many VC using non-parallel data, including zero-shot learning remains less explored areas in VC. Most of the many-to-many VC architectures require training data from all the target speakers for whom we want to convert the voices. In this paper, we propose a novel style transfer architecture, which can also be extended to generate voices even for target speakers whose data were not used in the training (i.e., case of zero-shot learning). In particular, propose Adaptive Generative Adversarial Network (AdaGAN), new architectural training procedure help in learning normalized speaker-independent latent representation, which will be used to generate speech with different speaking styles in the context of VC. We compare our results with the state-of-the-art StarGAN-VC architecture. In particular, the AdaGAN achieves 31.73%, and 10.37% relative improvement compared to the StarGAN in MOS tests for speech quality and speaker similarity, respectively. The key strength of the proposed architectures is that it yields these results with less computational complexity. AdaGAN is 88.6% less complex than StarGAN-VC in terms of FLoating Operation Per Second (FLOPS), and 85.46% less complex in terms of trainable parameters.   Language is the core of civilization, and speech is the most powerful and natural form of communication. Human voice mimicry has always been considered as one of the most difficult tasks since it involves understanding of the sophisticated human speech production mechanism (Eriksson & Wretling (1997) ) and challenging concepts of prosodic transfer (Gomathi et al. (2012) ). In the literature, this is achieved using Voice Conversion (VC) technique (Stylianou (2009) ). Recently, VC has gained more attention due to its fascinating real-world applications in privacy and identity protection, military operations, generating new voices for animated and fictional movies, voice repair in medical-domain, voice assistants, etc. Voice Conversion (VC) technique converts source speaker's voice in such a way as if it were spoken by the target speaker. This is primarily achieved by modifying spectral and prosodic features while retaining the linguistic information in the given speech signal (Stylianou et al. (1998) ). In addition, Voice cloning is one of the closely related task to VC (Arik et al. (2018) ). However, in this research work we only focus to advance the Voice Conversion. With the emergence of deep learning techniques, VC has become more efficient. Deep learningbased techniques have made remarkable progress in parallel VC. However, it is difficult to get parallel data, and such data needs alignment (which is a arduous process) to get better results. Building a VC system from non-parallel data is highly challenging, at the same time valuable for practical application scenarios. Recently, many deep learning-based style transfer algorithms have been applied for non-parallel VC task. Hence, this problem can be formulated as a style transfer problem, where one speaker's style is converted into another while preserving the linguistic content as it is. In particular, Conditional Variational AutoEncoders (CVAEs), Generative Adversarial Networks (GANs) (proposed by Goodfellow et al. (2014) ), and its variants have gained significant attention in non-parallel VC. However, it is known that the training task for GAN is hard, and the convergence property of GAN is fragile (Salimans et al. (2016) ). There is no substantial evidence that the gen-erated speech is perceptually good. Moreover, CVAEs alone do not guarantee distribution matching and suffers from the issue of over smoothing of the converted features. Although, there are few GAN-based systems that produced state-of-the-art results for non-parallel VC. Among these algorithms, even fewer can be applied for many-to-many VC tasks. At last, there is the only system available for zero-shot VC proposed by Qian et al. (2019) . Zero-shot conversion is a technique to convert source speaker's voice into an unseen target speaker's speaker via looking at a few utterances of that speaker. As known, solutions to a challenging problem comes with trade-offs. Despite the results, architectures have become more complex, which is not desirable in real-world scenarios because the quality of algorithms or architectures is also measured by the training time and computational complexity of learning trainable parameters ). Motivated by this, we propose computationally less expensive Adaptive GAN (AdaGAN), a new style transfer framework, and a new architectural training procedure that we apply to the GAN-based framework. In AdaGAN, the generator encapsulates Adaptive Instance Normalization (AdaIN) for style transfer, and the discriminator is responsible for adversarial training. Recently, StarGAN-VC (proposed by Kameoka et al. (2018) ) is a state-of-the-art method among all the GAN-based frameworks for non-parallel many-to-many VC. AdaGAN is also GAN-based framework. Therefore, we compare AdaGAN with StarGAN-VC for non-parallel many-to-many VC in terms of naturalness, speaker similarity, and computational complexity. We observe that AdaGAN yields state-of-the-art results for this with almost 88.6% less computational complexity. Recently proposed AutoVC (by Qian et al. (2019) ) is the only framework for zero-shot VC. Inspired by this, we propose AdaGAN for zero-shot VC as an independent study, which is the first GAN-based framework to perform zeroshot VC. We reported initial results for zero-shot VC using AdaGAN.The main contributions of this work are as follows: • We introduce the concept of latent representation based many-to-many VC using GAN for the first time in literature. • We show that in the latent space content of the speech can be represented as the distribution and the properties of this distribution will represent the speaking style of the speaker. • Although AdaGAN has much lesser computation complexity, AdaGAN shows much better results in terms of naturalness and speaker similarity compared to the baseline. In this paper, we proposed novel AdaGAN primarily for non-parallel many-to-many VC task. Moreover, we analyzed our proposed architecture w.r.t. current GAN-based state-of-the-art StarGAN-VC method for the same task. We know that the main aim of VC is to convert the source speaker's voice into the target speaker's voice while preserving linguistic content. To achieve this, we have used the style transfer algorithm along with the adversarial training. AdaGAN transfers the style of the target speaker into the voice of a source speaker without using any feature-based mapping between the linguistic content of the source speaker's speech. For this task, AdaGAN uses only one generator and one discriminator, which leads to less complexity. AdaGAN is almost 88.6% computationally less complex than the StarGAN-VC. We have performed subjective analysis on the VCTK corpus to show the efficiency of the proposed method. We can clearly see that AdaGAN gives superior results in the subjective evaluations compared to StarGAN-VC. Motivated by the work of AutoVC, we also extended the concept of AdaGAN for the zero-shot conversion as an independent study and reported results. AdaGAN is the first GAN-based framework for zero-shot VC. In the future, we plan to explore high-quality vocoders, namely, WaveNet, for further improvement in voice quality. The perceptual difference observed between the estimated and the ground truth indicates the need for exploring better objective function that can perceptually optimize the network parameters of GAN-based architectures, which also forms our immediate future work. At τ → ∞, the assumptions that made in Section 5.1 are true. Hence, from eq. (18), we can conclude that there exists a latent space where normalized latent representation of input features will be the same irrespective of speaking style. Theorem 2: By optimization of min En,De L C X→Y + L sty X→Y , the assumptions made in Theorem 1 can be satisfied. Proof: Our objective function is the following: Iterate step by step to calculate the term (t 2 ) used in loss function L sty X→Y . Consider, we have the latent representations S x1 and S y1 corresponding to the source and target speech, respectively. Step 1: S x1 (τ ) − µ 1 (τ ) σ 1 (τ ) σ 2 (τ ) + µ 2 (τ ) (Representation of t 1 ), Step 2&3: En De S x1 (τ ) − µ 1 (τ ) σ 1 (τ ) σ 2 (τ ) + µ 2 (τ ) . After applying decoder and encoder sequentially on latent representation, we will again get back to the same representation. This is ensured by the loss function L C X→Y . Formally, we want to make L C X→Y → 0. Therefore, we can write step 4 as: Step 4: S x1 (τ ) − µ 1 (τ ) σ 1 (τ ) σ 2 (τ ) + µ 2 (τ ) (i.e., reconstructed t 1 ), Step 5: 1σ 2 (τ ) S x1 (τ ) − µ 1 (τ ) σ 1 (τ ) ¨σ 2 (τ ) +¨μ 2 (τ ) −¨μ 2 (τ ) (Normalization with its own (i.e., latent representation in Step 4) µ and σ during AdaIN ), Step 6: S x1 (τ ) − µ 1 (τ ) σ 1 (τ ) (Final output of Step 5), Step 7: S x1 (τ ) − µ 1 (τ ) σ 1 (τ ) σ 1 (τ ) + µ 1 (τ ) (Output after de-normalization in AdaIN . Representation of t 2 ), where µ 1 and σ 1 are the mean and standard deviations of the another input source speech, x 2 . Now, using the mathematical representation of t 2 , we can write loss function L sty X→Y as: According to eq. (19), we want to minimize the loss function L sty X→Y . Formally, L sty X→Y → 0. Therefore, we will get µ 1 = µ 1 , and σ 1 = σ 1 to achieve our goal. Hence, mean and standard deviation of the same speaker are constant, and different for different speakers irrespective of the linguistic content. We come to the conclusion that our loss function satisfies the necessary constraints (assumptions) required in proof of Theorem 1.
Novel adaptive instance normalization based GAN framework for non parallel many-to-many and zero-shot VC. 
{"src": "scitldr-AIC_train", "paper_id": "HJlk-eHFwH"}