We are also interested in applying our data mixing laws to continual pretraining, which shares the same paradigm as pertaining but begins the model with pretrained parameters instead of random initialization. Generally, continual pretraining is a common technique to enhance existing pretrained models. It injects up-to-date knowledge into the model, avoiding performance degradation due to distribution shifts (Gururangan et al., 2020; Xiong et al., 2023). In addition, researchers also apply continual pretraining to reuse existing model parameters to build models of a different architecture (Komatsuzaki et al., 2022).