Pretraining data for large language models (LLMs) are typically a mixture of multiple domains, varying from English to minority languages (Doddapaneni et al., 2021; Li et al., 2023), from casual dialogs to formal academic writings (Taylor et al., 2022), and from texts to modalities like images and speeches (Zhan et al., 2024), among others. These data interplay with each other, showing complex interchangeable, unrelated, or contradictory relationships (Guo et al., 2024). This necessitates adjusting the mixture proportions of training data to balance the model capabilities while harnessing synergies across domains, thus enhancing the competence of the outcome models, as highlighted by extensive practical experience (Gao et al., 2020; Gururangan et al., 2020; Zhou et al., 2023; Xie et al., 2024).