All of the pre-trained word-level contextual embedding models include some character or subword components in their architecture. For example, Flair is a forward-backward LM trained over characters using recurrent neural networks (RNNs), that generates pre-trained contextual word embeddings by concatenating the forward LM's hidden state for the word's last character and the backward LM's hidden state for the word's first character. Flair achieves state-of-the-art or competitive results on part-of-speech tagging and named entity tagging<cite> (Akbik et al., 2018)</cite> .