We thus wonder whether we can obtain the losses of different mixture proportions without training at large scales. Fortunately, this idea gains endorsement from existing experiences that verify the impressive extrapolation of scaling laws of training steps and model sizes. In particular, OpenAI (2023) predicts the loss of the target model with merely \(1,000\times-10,000\times\) less compute. As a consequence, we can train small models with few training steps on different mixtures, and fitting scaling laws on them to estimate the losses of the target model size and the target number of training steps. We can then use the predicted losses to fit a data mixing law and search for the optimal mixture. The proposed pipeline is illustrated in Fig. 1 with details depicted in Alg. 1.