NotationGiven a source dataset, \(D_{source}\), of documents (crawled web pages) and model architecture, \(M\), we aim to find a strategy \(S\) for selecting a subset of these documents that maximizes some evaluation metric \(E(M(D_{S,R}))\). \(R\) indicates the proportion of remaining documents from the source dataset \(D_{source}\) after selecting data with strategy \(S\). For this reason, we refer to \(R\) throughout this work as the _selection ratio_: for example, if \(R=0.25\) and \(|D_{source}|=100\) million, then we _select_ 25% of documents from a source dataset of size \(100\)M documents to arrive at a a training dataset with \(25\)M documents. We operate at the granularity of a single document, independently of how the model trainer would pack these documents into batches later. Throughout the paper, we use random selection as the baseline for \(S\), as it is the most common method for selecting data for language model pre-training. In the rest of this section, we describe our choices of source dataset (\(D_{source}\)), model (\(M\)), evaluation metric (\(E\)), and, most importantly, our suggestions for the selection strategy (\(S\)).