Figure A20: Comparison of data selection strategies with the OPT model embedding space, when using D4 as a the selection strategy, when using C4 as the starting training dataset. The x-axis is selectoin ratio \(R\), and the y-axis is perplexity difference compared to baseline (the horizontal gray dotted line at 0.0 represents our baseline i.e. when no data selection is done), so **lower is better**. Notice that D4 and SemDeDup match at 90%, because we use \(R_{dedup}=0.9\) and varied \(R_{proto}\) for this experiment.