The primary pretraining corpus in our experiments is Colossal Clean Crawled Corpus (C4), a curated English text dataset comprising over 170 billion tokens. This corpus is derived from CommonCrawl, a common practice in the pretraining of LLMs Brown et al. (2020); Raffel et al. (2020); Touvron et al. (2023). This data source is also prominently featured in openly available LLM pretraining corpora, including The Pile Gao et al. (2020) and RedPajama Computer (2023). There are different versions of CommonCrawl data and our selection of C4 for pretraining is driven by driven by its size and quality.