In this section, we consider the fixed compute setting, where we curate and train on a fixed token budget by jointly increasing the size of the source dataset \(D_{source}\) and decreasing \(R\) (the fraction of the \(D_{source}\) which is selected), such that the target token budget remains constant. This setting is analogous to the most common paradigm for LLM training. As \(D_{source}\) grows and \(R\) decreases, we select from larger and larger initial datasets, resulting in a larger set of high-quality data points to select from and increasing the overall quality of the selected set. For clarity, we plot performance as a function of the ratio of the \(D_{source}\) to \(D_{target}\). For each setting, we evaluate the performance of a baseline, SemDeDup alone, SSL Prototypes alone, and our proposed method D4.