Commonly known as a cloze task,<cite> Devlin et al. (2018)</cite> introduced a framework that pretrained transformers (Vaswani et al., 2017) based on masked token prediction. Unlike<cite> Devlin et al. (2018)</cite> , we run the randomization script once per each training epoch. Otherwise, we follow the procedure in<cite> Devlin et al. (2018)</cite> .