Data selection.Data selection in continual pre-training plays a critical role in choosing the most valuable data samples for the training process. Various distributed and linguistic features independent of specific domains or tasks have been shown to be beneficial for data selection and the organization of learning curricula [21; 28]. In the context of LLMs, there is limited understanding of how to curate data for pre-training, let alone for continual pre-training.