D4 parameters: The choice of parameters \(R_{proto}\) and \(R_{dedup}\) while using D4 will have impact on the performance of D4. Given limited compute, we are not able to sweep over these hyperparameters. Instead, we strategically choose these parameters: we first look at the highest value of \(R\) in SemDeDup that results in perplexity improvement across validation sets. We choose the "highest value" because the purpose of SemDeDup is to remove duplicate-driven clusters and low \(R\) with SemDeDup generally removes more than just templates/semantic duplicates. As seen in Section A.3, this generally occured with \(R_{dedup}=0.75\). Thus, we chose \(R_{dedup}=0.75\) and varied \(R_{proto}\) to obtain different data selection ratios for D4.