Footnote 9: An idea is to transfer the optimized training mixture on small models trained with few tokens to the training of large models and large volumes of training data. Nevertheless, in Appendix A, we show that the partial order of data mixture concerning model performance varies as the model size and number of trained tokens change. Therefore, the optimal mixture at experimented scales can be suboptimal at the target scale.