UTTS: Unsupervised TTS with Conditional Disentangled Sequential Variational Auto-encoder
In this paper, we propose a novel unsupervised text-to-speech (UTTS)
framework which does not require text-audio pairs for the TTS acoustic modeling
(AM). UTTS is a multi-speaker speech synthesizer developed from the perspective
of disentangled speech representation learning. The framework offers a flexible
choice of a speaker's duration model, timbre feature (identity) and content for
TTS inference. We leverage recent advancements in self-supervised speech
representation learning as well as speech synthesis front-end techniques for
the system development. Specifically, we utilize a lexicon to map input text to
the phoneme sequence, which is expanded to the frame-level forced alignment
(FA) with a speaker-dependent duration model. Then, we develop an alignment
mapping module that converts the FA to the unsupervised alignment (UA).
Finally, a Conditional Disentangled Sequential Variational Auto-encoder
(C-DSVAE), serving as the self-supervised TTS AM, takes the predicted UA and a
target speaker embedding to generate the mel spectrogram, which is ultimately
converted to waveform with a neural vocoder. We show how our method enables
speech synthesis without using a paired TTS corpus. Experiments demonstrate
that UTTS can synthesize speech of high naturalness and intelligibility
measured by human and objective evaluations.