In recent years natural language processing has experienced rapid development. In particular, application of deep learning to NLP [1]}, the introduction of pre-trained context-independent word embeddings such as word2vec [2]}, [3]}, GloVe [4]} and fastText [5]}, and utilization of RNN-based [6]} pipelines such as CharCNN-BLSTM-CRF [7]}, [8]} allowed to improve SOTA for an overwhelming majority of NLP tasks. In 2018 a monumental breakthrough happened when context-dependent embeddings based on pre-trained language models emerged, such as ELMo [9]}, ULMFit [10]}, and BERT [11]}. The resulting boost in the models' performance cannot be underestimated, amounting to a 30% relative error reduction rate.
