Following Devlin et al. (2018) ;<cite> Lample and Conneau (2019)</cite> , in our CMLM objective, we randomly sample 15% of the BPE ngrams from the text streams, and replace them by [MASK] tokens 70% of the time.