Recently, pre-trained language models have been applied in many problems of NLP.
XLM-R [1]} is a mask language model built based on RoBERTa [2]} architecture. XLM-R was trained on the massive sizes of multilingual data.
BART [3]} is a denoising autoencoder for the sequence-to-sequence language model. BART is effective when fine-tuned for text generation task.
DeBERTa-V3 [4]} replaced mask language modeling (MLM) with replaced token detection (RTD) and implemented a gradient-disentangled embedding sharing method. DeBERTa-V3 achieves SOTA on many downstream tasks of NLP.
