The above distinctive characteristics throw several challenges in modeling L2M and M2L: 1) They require large amount of paired melody and lyric data to learn the mapping relationship between lyric and melody due to weak correlation. However, it is difficult to collect such large amount of paired data, and thus both tasks suffer from limited paired data; 2) They need additionally generate strict alignments between word/syllable in lyric and note in melody, and thus how to model the alignments well is critical to ensure the generation quality of lyric and melody. Previous works [1]}, [2]}, [3]}, [4]} on L2M and M2L have not considered the scenario of limited paired data, and only leverage some greedy decisions for lyric and melody alignment, which cannot well address these challenges. In this paper, we propose SongMASS, an automatic song writing system for L2M and M2L, which addresses the first challenge with masked sequence to sequence pre-training and the second challenge with attention based alignment constraint.
<FIGURE>