Data Optimal Regime.Unlike prior works that train language models in either "compute optimal regime" [HBM\({}^{+}\)22] or "over-train regime", we mainly focus on the quality of data for a _given scale_.2 We try to calibrate the training data to be closer to the "data optimal" regime for small models. In particular, we filter the web data to contain the correct level of "knowledge" and keep more web pages that could potentially improve the "reasoning ability" for the model. As an example, the result of a game in premier league in a particular day might be good training data for frontier models, but we need to remove such information to leave more model capacity for "reasoning" for the mini size models. We compare our approach with Llama-2 in Figure 2.