Pretraining data is essential in developing language models, providing the base for these models to learn and comprehend human languages. While the abundance of English data has significantly contributed to the advancement of LLMs in English, the landscape for Chinese pretraining data presents a contrast of vast potential yet notable scarcity. Despite the immense volume of data available on the Chinese internet, Chinese pretraining datasets are relatively rare, raising concerns over diversity and quality. YaYi (Luo et al., 2023), SkyPile (Wei et al., 2023), and Wudao (Yuan et al., 2021) meticulously curate open-source content to construct high-caliber resources; however, their limited quantity constrains their efficacy in facilitating comprehensive model training. Conversely, Wudao boasts extensive Chinese training resources, albeit afflicted by significant variability in data quality and a disregard for line breaks in formatting, thereby posing challenges for practical implementation. ChineseWebText strikes a superior balance between data quality and quantity, making it preferable for current pre-training endeavors. Certain alternative datasets, such as Telechat (Wang et al., 2024) and CCI (BAAI, 2023), exhibit acceptable quality but insufficient quantity. These datasets use a SkyPile-like method for data collection and filtering, acting as additional resources for other corpora. Furthermore, although COIG series (Zhang et al., 2023; Zheng et al., 2024) is categorized as SFT data, it holds promise for large-scale pre-training applications due to its vast volume. Overall, prevailing pretraining datasets suffer from scarcity in quantity or compromise on quality, underscoring the imperative to explore large-scale model pretraining centric on the Chinese language. Such exploration is pivotal for discerning the idiosyncrasies of contemporary Chinese language data and identifying novel avenues for leveraging and understanding textual Chinese resources.