Determining Data Mix RatioThe quality of pre-training datasets is the one of the most important factors that affects the quality of large language models [15, 16]. However, determining the optimal mix ratio from multiple data sources is an extremely challenging task as it involves combinatorial combinations [13]. Existing works usually determine domain weights (the sampling probabilities for each domain) by using intuition or a set of downstream tasks. Scaling laws can offer some new insights in helping determine the optimal mix ratio. By predicting the test loss trajectory of large models on each individual data source, we could implicitly infer how important and useful each data source is (_e.g._, if the loss decreases faster in one data source and converges into a lower loss value, then this data source might be more useful).