DeduplicationAfter filtering, we implement a comprehensive deduplication pipeline following the procedure in Penedo et al. (2023) [56]. This pipeline integrates document-level MinHash deduplication and sub-document exact-match deduplication, effectively identifying and removing duplicate content within and across documents. We further categorize web documents into specific themes using a topic model predicting labels like as news, ads, and knowledge-based content. In the final pretraining dataset, we down-sample less helpful content, mostly advertisements, to ensure information density. The final composition of Yi's pretraining data is shown in Fig. 2.