All the models are pre-trained on a mixture of English and Chinese corpus. Both the English and Chinese corpora consist of webpages, wikipedia, books, and papers. The ratio of English to Chinese is 4:1 in the pre-training corpus. We tokenize the data with the byte pair encoding (BPE) algorith [38] with the SentencePiece package [22].