Colossal Cleaned Common Crawl (C4)is made up of 360M web documents, with an average length of 486 BPE tokens Raffel et al. (2020). C4 was introduced as a pre-training dataset for T5, a set of encoder-decoder models which have been widely used in fine-tuned downstream tasks. The dataset was previously deduplicated in a more sophisticated process than the prior two datasets. Each paragraph was hashed and paragraphs resulting in hash collisions were removed. This was followed by a pass that removed placeholder text, code, and prohibited words. See Dodge et al. (2021) for a detailed breakdown of the source text in C4.