We choose Transformer [1]} as our basic model structure, which consists of 6 encoder/decoder layers. The hidden size and filter size of each layer are set as 512 and 2048. The number of attention heads is 8. We use the same masking strategy as in  [2]}. We use Adam optimizer [3]} with a learning rate of 5e-4. The model is trained on a NVIDIA Tesla T4 GPU card, and each mini-batch contains 4096 tokens. During training, we apply dropout with the rate of 0.1. The hyper-parameter \(\alpha \)  is set as 0.5. The dataset is split as training/valid/test set with a ratio of 8:1:1. Our baseline is a standard Transformer model, using the same model configuration with SongMASS but without any pre-training or alignment constraints.
