Specifically, to handle the challenges of limited paired data, we leverage self-supervised pre-training on large amount of unpaired lyric and melody data. Since L2M and M2L are both sequence to sequence learning tasks, we adopt masked sequence to sequence pre-training (MASS) [1]}, which is a popular pre-training method by masking a segment in the input sequence and predicting this segment in the output using an encoder-decoder framework. However, simply using original MASS in L2M and M2L cannot well handle the long lyric and melody sequence in song level and the diversity between lyric and melody modality. Therefore, we introduce two extensions on MASS: 1) Instead of masking in a single sentence in original MASS, we design a song-level masked pre-training strategy to capture longer contextual information, since music usually has repeat structure and relies on long context. 2) Unlike original MASS, we use separate encoder-decoder for lyric-to-lyric and melody-to-melody masked pre-training since they are in different modalities. However, separate training of lyric-to-lyric and melody-to-melody cannot ensure to learn a shared latent space between lyric and melody and thus could harm the transformation between them. Therefore, we add supervised training with paired lyric and melody data to guide the pre-training towards learning a shared latent representation between lyric and melody modality.
