With models such as ELMo (Peters et al., 2018) , GPT-2 (Radford et al., 2018) , and BERT<cite> (Devlin et al., 2018)</cite> , important progress has been made in learning improved sentence representations with context-specific encodings of words via a language modeling objective. The latter two approaches both rely on Transformer encoders, but BERT is trained using masked language modeling instead of right-to-left or left-to-right language modeling. Additionally, BERT also optimizes a next sentence classification objective.