Pre-trained Language models have been proved to be essential while dealing with sequence-to-sequence task. We present the language model we utilized in this section and demonstrate our consideration. We use RoBERTa [1]} for language comprehension task. The structure of RoBERTa is based on BERT [2]}. We also use BART [3]} for generation task. BART is a denoising autoencoder for pre-training sequence-to-sequence models. It uses the standard sequence-to-sequence Transformer architecture except the activation functions which is replaced by GeLUs. We find the structure of autoregressive decoder make the model can be directly ﬁne tuned for sequence generation tasks. BART is also enable to apply any type of document corruption which make the additional knowledge available to the model.
