Using pre-trained word embeddings for feature extraction has been shown to be highly effective in multiple NLP tasks. Traditional word embeddings are extracted from shallow neural networks trained on a large swathes of texts required to learn the contextual representations of words. Examples include skip-grams [1]} and GloVe [2]}. However, these embeddings are learned from an aggregation of all possible word contexts, which may gloss over semantic nuances in representations.
