To evaluate different configurations of data selection strategies, we train OPT [59] models from scratch on the pruned versions of datasets. We use the standard model architectures and settings of Zhang et al. [59] and use MetaSeq [59] to train all our models. For 125M models, we train to \(T_{target}=3B\) tokens. For 1.3B parameter models, we train to target token count of \(T_{target}=40B\). For 6.7B parameter models, we train to \(T_{target}=100B\) tokens. We choose these by trimming down the token budgets suggested by Hoffmann et al. [20] to meet our compute limitations. We provide full details of our training setup in Section A.1.