Thus contextual language representation based on pre-trained models including CoVe (McCann et al., 2017) , ELMo (Peters et al., 2018) , GPT (Radford et al., 2018 (Radford et al., , 2019 and BERT<cite> (Devlin et al., 2019)</cite> becomes prevalent recently.