In this subsection, we describe how we curate _CC-dedup_, the starting source dataset used throughout the paper. We start with 5 CommonCrawl dumps 2 which range from 2017 to 2020. We then use CC-net [54], to de-duplicate data at the paragraph level, remove non-English web pages, and filter out low-quality pages. The pipeline we use is identical to the pipeline used in Touvron et al. [50] (see the section after the subtitle "English CommonCrawl [67%]", within Section 2).