Pretraining Data OptimizationThe objective of optimizing pre-training corpora is to maximize the performance and efficiency of language model training by improving the quality and scale of the pretrain data mixture. This typically includes data collecting through crawling (Raffel et al., 2020) or synthesis (Polu and Sutskever, 2020; Gunasekar et al., 2023), de-duplication (Lee et al., 2021; Kandpal et al., 2022; Tirumala et al., 2023), filtering and selection (Albalak et al., 2024), as well as data composition (Xie et al., 2024) and curriculum (Chen et al., 2024; MA et al., 2024).