Pre-trained language models (LMs) such as ELMo (Peters et al., 2018) , ULMFiT (Howard and Ruder, 2018) , OpenAI GPT (Radford et al., 2018) , Flair<cite> (Akbik et al., 2018)</cite> and Bert (Devlin et al., 2018) have shown great improvements in NLP tasks ranging from sentiment analysis to named entity recognition to question answering.