When evaluating on the validation sets of LM1B and Wiki-40B, we found that models trained on NearDup-deduplicated C4 consistently achieved lowest perplexity (for LM1B eval with base models, see Appendix Figure 7).