RealNewsis a subset of the Common Crawl consisting of articles from news domains Zellers et al. (2019). It contains 31M documents with average length 793 BPE tokens. RealNews was deduplicated by inserting a hash of the first 100 characters of each document into a bloom filter Bloom (1970) and then excluding any document which resulted in a hash collision. Like C4, examples with duplicate URLs were excluded.