Wikipedia (Wiki-40B)consists of multi-lingual cleaned Wikipedia text Guo et al. (2020). We take the English portion, which contains 2.9M Wikipedia pages with an average length of 768 BPE tokens. The dataset creators do not indicate any deduplication was performed aside from removing redirect-pages (e.g., "sunflower" to "Helianthus").