**CommonCrawl.** CommonCrawl [163] is one of the largest open-source web crawling databases, containing a petabyte-scale data volume, which has been widely used as training data for existing LLMs. As the whole dataset is very large, existing studies mainly extract subsets of web pages from it within a specific period. However, due to the widespread existence of noisy and low-quality information in web data, it is necessary to perform data preprocessing before usage. Based on CommonCrawl, there are four filtered datasets that are commonly used in existing work: C4 [82], CC-Stories [155], CC-News [27], and RealNews [156]. The Colossal Clean Crawled Corpus (C4) includes five variants21, namely en (806G), en.noclean (6T), realnewslike (36G), web-textlike (17G), and multilingual (38T). The _en_ version has been utilized for pre-training T5 [82], LaMDA [68], Go-pher [64], and UL2 [89]. The multilingual C4, also called mC4, has been used in mT5 [83]. CC-Stories (31G) is composed of a subset of CommonCrawl data, in which the contents are made in a story-like way. Because the original source of CC-Stories is not available now, we include a reproduction version, _CC-Stories-R_[164], in Table II. Moreover, two news corpora extracted from CommonCrawl, _i.e._, REALNEWS (120G) and CC-News (76G), are also commonly used as the pre-training data.