Our findings also hold at larger model scales. We pick our best-performing configuration from 1.3B OPT experiments (e.g., \(R=0.25\)) and train 6.7B OPT models on 100B tokens. Figure 1 shows the positive effects of applying D4 with \(R=0.25\) for a 6.7B model. The model trained on the pruned data reaches the same perplexity as the baseline model using 20% fewer update steps on average and achieves a 2% improvement in accuracy on our suite of downstream tasks at the end of the training - about as much difference as was reported by Zhang et al. [59] between the OPT and GPT-3 family of models on the same set of tasks (See Figure 3 of Zhang et al. [59]).