We introduce our training data distribution in Figure 11. In the figure, CommonCrawl_Chn in a Chinese Corpus is derived from CommonCrawl raw corpus and goes through thorough cleaning. Dolma (Soldaini et al., 2024), C4 (Raffel et al., 2019), and Pile (Gao et al., 2020; Biderman et al., 2022) are English corpora. They are deduplicated inner corpus and across corpus using MinHash algorithms (Broder, 1997). The Code Pre-train data contains the stack (Kocetkov et al., 2022) and StarCoder Li et al. (2023), with inner deduplication and cross deduplication. In the decay stage, the data mixture contains more diverse data and proprietary data, including UltraChat (Ding et al., 2023), SlimOrca (Lian et al., 2023;3b), OssInstruct (Wei et al., 2023), EvolInstruct (Xu et al., 2023). The data with the suffix SFT is our proprietary data including LectCode questions, Kindergarten through 12th grade (K12) textbooks and questions, etc.