In terms of Vietnamese language modeling, to the best of our knowledge, there are two main concerns: (i) The Vietnamese Wikipedia corpus is the only data used to train all monolingual language models , and it also is the only Vietnamese dataset included in the pre-training data used by all multilingual language models except XLM-R<cite> [Conneau et al., 2019]</cite> .