BART(Lewis et al., 2019) is an Encoder-Decoder model that allows for arbitrary noise transformations, as the input to the encoder does not need to align with the output of the decoder. In this case, the document is corrupted by replacing the text span with mask symbols. For pre-training, the researchers proposed five models: Token Masking, Token Deletion, Text Infilling, Sentence Permutation, and Document Rotation. For fine-tuning, the encoder and decoder are fed an uncorrupted document, and the representation of the final hidden state from the decoder is used. Many researchers(Thulke et al., 2021) (Siriwardhana et al., 2023) (Lewis et al., 2019) have adopted BART as the language model in the RALM architecture due to its comprehensive and novel pre-training approach, which greatly enhances the model's robustness.