Speech representation encoders:
Inspired by SpeechSplit [1]}, the rhythm encoder \(E_r\) , pitch encoder \(E_p\) , content encoder \(E_c\)  have the same structure, which
consists of a stack of 5 × 1 convolutional layers and a stack of bidirectional long short-term memory (BiLSTM) layers.
Besides mel-spectrogram, the pitch contour also carries information of rhythm [2]}.
Before the input mel-spectrogram \({S}\)  and normalized pitch contour \({P}\)  are fed to the content encoder and pitch encoder, random resampling [3]} operation along the time dimension is performed to remove rhythm information.
The model only relies on the rhythm encoder to recover the rhythm information.
The speaker encoder [4]} \(E_t\)  is used to provide global speech characteristics to control the speaker identity.
Denote the random resampling operation as \(RR(\cdot )\) , then we have:
\(\begin{split}{Z}_{r}=E_{r}({S}), \quad {Z}_{c}=E_{c}(RR({S})), \\{Z}_{t}=E_{t}({S}), \quad {Z}_{p}=E_{p}(RR({P})) \\\end{split}\) 
