The overall architecture of SongMASS for L2M and M2L is shown in Figure REF , which adopts the Transformer [1]} based encoder-decoder framework. We employ separated encoders and decoders for lyric and melody respectively due to the large diversity between lyric and melody. To leverage the knowledge from large-scale unlabeled lyrics or melodies, we perform MASS pre-training for lyric-to-lyric and melody-to-melody in our framework. We pre-train our model in song level to better capture long contextual information for lyric or melody sequence, and incorporate supervised learning (lyric-to-melody and melody-to-lyric) into our pre-training to learn a shared latent space between different modalities. To learn the alignment between word/syllable in lyric and note in melody, we further leverage sentence-level constraint and token-level constraint into our model to guide the alignment between lyric and melody. We use a dynamic programming strategy to obtain the final strict alignment between the lyric and melody. In below, we describe the details of our pre-training methods and alignment strategies in SongMASS.
<FIGURE><FIGURE>