Figure 1: Pretraining data distribution, where "zh" represents Chinese data, “en” represents English data, “cc” stands for Common Crawl, including publicly available web documents, etc., and ‘encyc.’ refers to the encyclopedia.