Few-shot voice cloning alone is a challenging task, due to the reason that the trade-off between model capacity and overfitting issues must be well dealt with.
Existing approaches can be roughly classified into two kinds [1]}, i.e., speaker encoding approach and speaker adaptation approach, in terms of whether to update parameters of the base TTS model during voice cloning.
Speaker encoding approach does not update parameters of the base TTS model and represents speaker characteristics using speaker embedding vectors, which are expected to control the output voice. This approach does not involve a fine-tuning process and conducts voice cloning in a zero-shot adaptation manner. In [2]}, a speaker encoder model is first separately trained with a speaker verification loss [3]} leveraging large-scale low-quality data; and then adopted to compute speaker embedding vectors for a multi-speaker TTS model. During voice cloning, the speaker encoder computes an embedding vector from the limited adaptation data for a new speaker and adapts the base TTS model to the new voice by using the embedding vector as conditioning. Instead of pre-training a speaker verification model using external data, the speaker encoder model can also be trained using the TTS data, either jointly optimized with the base TTS model or separately optimized [1]}, [5]}, [6]}. This kind of approaches well tackle the possible overfitting issue since there is no finetuning process. However, embedding vectors computed by a pre-trained speaker encoder may not generalize well to an unseen speaker, which further degrades the voice cloning performance.
Speaker adaptation approach [1]}, [5]}, [9]} finetunes all or part of the base TTS model with limited adaptation data, which rely on careful selecting weights that are suitable for adapting and adopting proper adaptation scheme to update these weights in order to avoid overfitting the adaptation data. However, the cloning speed is much slow than the speaker encoding approaches [9]}.
