During the pre-training phase, we follow the common multi-speaker training procedure with loss function in Eq. (2), where we set \(\alpha _1=0.01\)  and \(\alpha _2=0.02\) .
In this work, we focus on 5-shot (around 12 second speech data in total) style transfer.
Therefore, during the meta learning phase, we sample 5 training samples and 5 test samples from one speaker to form a meta task. We regard a pair of speaker-prosody as a pseudo-speaker. Thus, in total there are 1400 pseudo-speakers, which are split into a meta training set (1300 pseudo-speakers) and a meta validation set (100 pseudo-speakers). We randomly initialize a speaker vector before meta learning, i.e., the speaker LUT only has one shared entry across all pseudo-speakers.
We use a meta batch size of 10. Meta learning rate is 0.0001 and base learning rate is 0.001. The Adam optimizer [1]} is adopted as the meta optimizer, while the vanilla SGD optimizer is adopted as the base optimizer.
