**Validation Perplexity.** In Figure 2, we show that a relatively small amount of data selection using any of the three methods (small \(R\)) brings consistent improvements on all validation sets. However, as we increase \(R\), we observe _opposing effects_ on web snapshot and non-web-snapshots validation sets. We analyze this discrepancy in-depth in Section 4.4. However, on the Instruct OPT validation set, which corresponds much more closely to the the high-quality generations we want our LLMs to achieve, we found that all three methods led to consistent and clear perplexity improvements. Notably, we found that while all three methods provided benefits, D4 outperformed using both SemDeDup and SSL Prototypes independently, with the most notable gains exhibited when the source dataset is around 4x the target dataset size. Given that D4 consistently improves with source dataset size, we estimate this gap to grow with source dataset size.