Automatic Evaluation of Speaker Similarity
We introduce a new automatic evaluation method for speaker similarity
assessment, that is consistent with human perceptual scores. Modern neural
text-to-speech models require a vast amount of clean training data, which is
why many solutions switch from single speaker models to solutions trained on
examples from many different speakers. Multi-speaker models bring new
possibilities, such as a faster creation of new voices, but also a new problem
- speaker leakage, where the speaker identity of a synthesized example might
not match those of the target speaker. Currently, the only way to discover this
issue is through costly perceptual evaluations. In this work, we propose an
automatic method for assessment of speaker similarity. For that purpose, we
extend the recent work on speaker verification systems and evaluate how
different metrics and speaker embeddings models reflect Multiple Stimuli with
Hidden Reference and Anchor (MUSHRA) scores. Our experiments show that we can
train a model to predict speaker similarity MUSHRA scores from speaker
embeddings with 0.96 accuracy and significant correlation up to 0.78 Pearson
score at the utterance level.