Due to computational limits, initial work on language model pre-training focused on training models on small, high-quality text datasets such as BookCorpus [61] and Wikipedia [32]. More recently, however, catalyzed by works like [40], advancements in large language models (LLMs) have been driven by leveraging large collections of unlabeled, uncurated data derived from snapshots of the internet (CommonCrawl [16; 39; 41]), trading off small quantities of heavily-curated data for huge quantities of less-curated data. Because of the dramatic increase in data quantity, these strategies have resulted in higher performance models and have sparked a new paradigm wherein massive, largely unfiltered datasets are utilized for training [11; 46; 50].