Unpaired Lyric and Melody. We use “380,000+ lyrics from MetroLyrics"https://www.kaggle.com/gyani95/380000-lyrics-from-metrolyrics as our unpaired lyrics for pre-training, which contains 362,237 songs. The lyrics in each song are split into sentences by the line break. For unpaired melodies, we choose “The Lakh MIDI Dataset" [1]}https://colinraffel.com/projects/lmd. The dataset contains 176,581 MIDI files with complete tracks, and we extract the melody tracks by Midi-minerhttps://github.com/ruiguo-bio/midi-miner. Finally, we get 65,954 melodies as our unpaired data for pre-training.
According to the characteristics of vocal melody, we consider the pitch and duration tokens of each note as the melody sequence. Each melody is transposed to the scale of C major or A minor. All the notes are shifted by octave so that the most pitches of the song fall into one-lined octave (MIDI pitch from 60 to 71). For unpaired melody MIDI file, we calculate the starting beat and duration of the note based on the absolute time and the BPM (Beats Per Minute), all the notes are aligned to 1/16 notes as paired data. We spread the melodies into sequences of pitch-duration patterns, as melody sequences for our model. For example, the melody in Figure REF  will be represented as “R, 7/16, G3, 1/16, E4, 1/8 ...". During pre-training, we simply split the unpaired melodies into phrases according to the average phrase length in paired data, since there is no natural phrase segmentation symbol in the MIDI files.
Paired Lyric and Melody. We use the LMD dataset [2]}https://github.com/yy1lab/Lyrics-Conditioned-Neural-Melody-Generation which contains aligned melodies and lyrics from 7,998 songs. We apply the same operation, as aforementioned, to process melody and lyric data. The lyrics/melodies are split into sentences/phrases based on the annotations.
