This had lead to the development of pre-trained systems like Bidirectional Encoder Representations from Transformers (BERT) [1]}, where pre-trained representations capture a large spectrum of contexts and generate embeddings of each word based on the underlying context in the sentence. RoBERTa [2]} is an improvement to the BERT model using larger training data with longer sequences, and dynamic masking. It has achieved achieve state-of-the-art performance on Multi-Genre Natural Language Inference Corpus (MNLI) [3]}, Question-answering NLI (QNLI) [4]}, Semantic Textual Similarity Benchmark (STS-B) [5]}, ReAding Comprehension dataset from Examination (RACE) [6]} and GLUE [4]}.
