Distributed representations of words, as in Word2Vec [1]}, GloVe [2]} and FastText [3]}, have brought significant improvements to NLP. Contextualized word embeddings such as ELMo [4]} and Flair [5]} have made it possible to provide more context-sensitive (hence more accurate) representations of words, and a growing list of embeddings [6]} followed. More recently, models exploiting a self-supervised objective with masking such as BERT [7]} have brought even more powerful representations. Better hyper-parameter optimization, such as in RoBERTa [8]}, have resulted in further improvements. Multilingual versions of MLMs such as mBERT and XLM-Roberta [9]} were also trained. Other models with different objectives and/or architectures such as ALBERT [10]}, T5 [11]} and GPT3 [12]} were also introduced. We briefly describe each of these models next.
