At the end of pretraining data generation process, we accumulated near 3.5 million samples, only running the process once on our train split, so without any duplication unlike <cite>Devlin et al. (2018)</cite> because of time restrictions.