Figure A17: Perplexity (y-axis) versus selection ratio \(R\) (x-axis) for different embedding spaces, when selecting data via D4. Across different 8m (top), 125m (middle) and 1.3b (bottom) model scales, we see that the SentenceTransformer embedding spaces outperform the OPT embedding space, but at the 6.7b model scale, we see that the OPT embedding space begins outperforming the all Mini LM v6 embedding space. We were unable to run an ”all-mp-net-base-v2” 6.7b experiment due to compute restrictions, but we note that the difference between ”all-mini-lm-v6” and ”all-mp-net-base-v2” across model scales and selection ratios is generally small, so we expect the OPT embedding space to outperform the ”all-mp-net-base-v2” at the 6.7b scale.