We also compare with pre-training on the Refined Web corpus Penedo et al. (2023). The dataset is also derived from the CommonCrawl, however has a more stringent filtering process. Our selection of Refined Web is for comparing synthetic rephrases to high quality subsets of web data, which were shown to achieve similar performance compared with curated datasets Penedo et al. (2023). For our experiments we used the first 3050 files and train for 300B tokens to match training on C4. We also conduct experiments with the first 1650 files to account for multiple epochs on the Refined Web dataset.