We perform all of our training runs on a version of CommonCrawl pre-processed with a CCNet [54] pipeline identical to the one used by Touvron et al. [50]. We add an additional step of MinHash-based de-duplication (see more details in Section A.1). Applying this common step before our experiments guarantees that any effects observed in our experiments complement the currently prevalent approach of MinHash-based data de-duplication strategies. Throughout the rest of this work, we refer to this dataset as _CC-dedup_.