Figure 2: Yi’s pre-training data mixture. Overall our data consist of 3.1T high-quality tokens in Both English and Chinese, and come from various sources. Our major differences from existing known mixtures like LLaMA [76] and Falcon [56] are that we are bilingual, and of higher quality due to our more rigorous cleaning pipeline.