Figure 2: Impact of deduplicating the training set on validation perplexity. We plot the results from T5 XL (see Appendix for base-sized model). For C4, we evaluate on _C4 Original_, the original validation set; _C4 Unique_, a subset of the validation set identified by NearDup as having zero matches across C4; and _C4 Duplicates_, a subset of the validation set identified by NearDup as having a match in the C4 train set.