Small-scale study.We first consider popular public web datasets (OSCAR-2019 (Ortiz Suarez et al., 2019), OSCAR-2022 (Abadji et al., 2021), C4 (Raffel et al., 2020)), The Pile (Gao et al., 2020) as the most popular publicly available curated dataset, and variations of RefinedWeb (RW-Raw, RW-Filtered, and RW as described in Section 3). For this first study, all models are trained with the same architecture and the same internal codebase; they are also all evaluated within the same framework-only pretraining datasets differ.