The architecture of the timbre encoder follows [1]}, which is composed of a 2-D convolution stack, a GRU and a style token layer. The input to the timbre encoder is logarithmic Mel-spectrogram.
The pre-trained VC model used for timbre augmentation follows [2]} and is trained on LibriSpeech [3]} dataset.
Both the target decoder and auxiliary decoder are a 3-layer Conformer with a hidden size of 512. The auxiliary encoder is composed of 4 convolutional residual blocks with hidden sizes of 128, 128, 128, 128 and 16.
Each convolutional residual block contains 2 convolution layers followed by LeakyReLU activation.
