The training of large language models (LLMs)  is expensive. In this paper, we study data-efficient approaches for pre-training LLMs,i.e., techniques that aim to optimize the Pareto frontier of model quality and training resource/data consumption. We seek to understand the tradeoffs associated with data selection routines based on (i)  expensive-to-computedata-qualityestimates, and (ii)  maximization ofcoverageand diversity-based measures in the feature space. Our first technique,Ask-LLM, leverages the zero-shot reasoning capabilities of instruction-tuned LLMs to directly assess the quality of a training example. To target coverage, we proposeDensitysampling, which models the data distribution to select a diverse sample. In our comparison ofsamplers, involving hundreds of evaluation tasks and pre-training runs, we find thatAsk-LLMandDensityare the best methods in their respective categories. Coverage sampling canrecoverthe performance of the full data, while models trained onAsk-LLMdata consistentlyoutperformfull-data training—even when we reject% of the original dataset, while converging up to% faster.