GeneralizationIn math continual pretraining, as depicted in Figure 6, training exclusively with SLM leads to quickly convergence to the domain focused by the reference model, accompanied by a significant rise in the loss of unselected tokens. Although no adverse effects, like biases, have been observed from the increased loss yet, a general pretraining loss on text and code may prevent overfitting Goodhart and Goodhart (1984), as suggested by Ouyang et al. (2022) and Azerbayev et al. (2023). Furthermore, future efforts could broaden the corpus scope of the reference model, and enlarge the pretraining data size, as exemplified by DeepSpeedMath Shao et al. (2024).