\(\bullet\)_Increasing the diversity of data sources._ Recent studies have empirically shown that training on excessive data about a certain domain would degrade the generalization capability of LLMs on other domains [35, 64]. In contrast, increasing the data source heterogeneity (_e.g.,_ including diverse data sources) is critical for improving the downstream performance of LLMs [229, 230, 212]. To further examine the effect of different data sources, some studies have conducted ablation experiments by removing each data source one by one, and pre-train LLMs with specially curated datasets [212]. It has been shown that dropping data sources with high heterogeneity (_e.g.,_ webpages) impacts LLM's abilities more severely than dropping sources with low heterogeneity (_e.g.,_ academic corpus).