Pre-trained language models (e.g., BERT [1]}, GPT [2]}, XLNet [3]}, MASS [4]} and etc) have achieved significant progress in natural language processing. They usually employ specific self-supervised tasks and pre-train on large-scale unlabeled data corpus to improve the understanding and generation capability. MASS [4]} is the first and one of most successful pre-training methods for sequence to sequence learning tasks, and several pre-training methods such as BART [6]} and T5 [7]} are also proposed to handle this kind of task. In this paper, we build our pre-training method upon MASS considering its popularity for sequence to sequence learning tasks and suitability for different modalities. Given a sequence from the unpaired sentence corpus, MASS randomly replaces a segment of tokens with mask tokens and takes the masked sequence as the encoder input and predicts the masked segment in the decoder. We leverage the basic idea of MASS and extend it with several improvements to address the distinctive challenges in the pre-training of L2M and M2L.
