A second method of speaker adaptation performs zero-shot adaptation via speaker embeddings [1]}, [2]}. We use this approach in our TBVE model. This involves training a TTS system to reconstruct the speech of multiple speakers, with the help of embeddings obtained from a pretrained speaker verification system. At test time, speaker embeddings can be extracted from target speakers not in the training corpus in order to perform zero-shot speaker adaptation. The downside of this approach however is that the speaker characteristics often are not an exact match for any given target speaker since fewer model parameters are adapted than finetuning. In this work we counteract this weakness by adopting a reference encoder that is trained jointly with the TBVE task and as such can capture details within an utterance not captured by the speaker verification model that are beneficial for voice cloning and therefore also voice editing.
