Various pre-training tasks were explored including traditional NLP tasks like machine translation (Mc-Cann et al., 2017) and language model (Peters et al., 2018; Radford et al., 2018 Radford et al., , 2019 , or other tasks such as masked language model and next sentence prediction<cite> (Devlin et al., 2019)</cite> .