Out-of-Domain Test Loss PredictionFor the out-of-domain test set, we use a private Chinese data whose type is very rare in the training data and can be considered as out-of-domain data. The estimated constant terms, together with the actual and predicted loss trajectories of 500M, 2B and 33B models using the estimated formulas are depicted in Figure 3. It is evident that predicting out-of-domain data is more challenging than predicting in-domain data, as the actual loss trajectory exhibits significant fluctuations. Nonetheless, the overall trend of actual and predicted loss trajectories closely aligns. The final converged loss values are also rather similar, affirming the efficacy of scaling laws in predicting the loss trajectory for both in-domain and out-of-domain data.