In this paper, we propose adversarial mutual information learning based SRD for one-shot VC.
Specifically, we first decompose a speech into four factors: rhythm, pitch, content and timbre.
Then we propose a system consisting of four components:
(1) Random resampling [1]} operation along the time dimension is performed on pitch and content to remove rhythm information;
(2) A common classifier is used for timbre to extract features related to speaker identity, and a gradient reversal layer (GRL) based adversarial classifier is used for speaker-irrelevant information (content, pitch and rhythm) to eliminate speaker information in latent space;
(3) The variational contrastive log-ratio upper bound (vCLUB) of mutual information (MI) [2]} between different representations is used to separate speaker-irrelevant information from each other;
(4) A pitch decoder is used to reconstruct normalized pitch contour and a speech decoder is adopted to reconstruct mel-spectrogram.
Main contributions of our work are:
(1) Implementing one-shot VC with decoupling content, timbre, rhythm and pitch from the speech,
which ensures pitch, rhythm and content more speaker-irrelevant and makes timbre more relevant to speaker.
(2) Applying SRD with vCLUB and GRL based adversarial mutual information learning without relying on text transcriptions.
Experiments show the information leakage issues can be effectively alleviated by applying with adversarial mutual information learning.
