We use the same large-scale data which has been used for BERT model pre-training, the BooksCorpus (800M words) (Zhu et al., 2015) and English Wikipedia (2.5B words) (Wikipedia contributors, 2004; <cite>Devlin et al., 2018)</cite> .