When selecting data for training large-scale models, standard practice is to
filter for examples that match human notions of data quality. Such filtering
yields qualitatively clean datapoints that intuitively should improve model
behavior. However, in practice the opposite can often happen: we find that
selecting according to similarity with “high quality” data sources may not
increase (and can evenhurt)  performance compared to randomly selecting
data. To develop better methods for selecting data, we start by framing dataset
selection as an optimization problem that we can directly solve for: given
target tasks, a learning algorithm, and candidate data, select the subset that
maximizes model performance. This framework thus avoids handpicked notions of
data quality, and instead models explicitly how the learning process uses train
datapoints to predict on the target tasks. Our resulting method greatly improves
language model (LM)  performance on both pre-specified tasks andpreviously unseentasks. Specifically, choosing target tasks
representative of standard LM problems and evaluating on diverse held-out
benchmarks, our selected datasets
provide a 2× compute multiplier over baseline methods.