In this section, we describe the architecture of Meta-Voice for multi-speaker and multi-style speech generation.
As shown in Fig. REF , Meta-Voice consists of four major modules: a text encoder, a variance adaptor, a mel decoder and a style encoder.
In this work, we decouple a speaking style into speaker and prosody components, omitting other factors such as phonetics and channel effects, according to the “subtractive" definition of prosody in [1]}.
We extend the style-adaptive layer normalization (SALN) presented in [2]} to support multiple speakers and prosodic types and add SALN layers in both the text encoder and the mel decoder, making Meta-Voice fully conditioned on speaking style information.
We split parameters \(\theta \)  in Meta-Voice into shared parameters \(\bar{\theta }\) , speaker-related parameters \(\theta ^{s}\)  and prosody-related parameters \(\theta ^{p}\) , i.e., \(\theta =\lbrace \bar{\theta }, \theta ^s, \theta ^p\rbrace \) .
\(\bar{\theta }\)  contains parameters of the text encoder, the variance adaptor and the mel decoder, which are responsible of producing right pronunciations corresponding to the input textual content. \(\theta ^{s}\)  and \(\theta ^{p}\)  as a whole contain parameters of the style encoder, which generates scaling and shifting parameters for SALN layers, modulating hidden features in both the text encoder and mel decoder to control timbre and prosodic characteristics of generated speech.
