**Balancing the Data Distribution.** Since instruction tuning involves a mixture of different tasks, it is important to balance the proportion of different tasks during fine-tuning. A widely used method is the _examples-proportional mixing_ strategy [82], _i.e.,_ combining all the datasets and sampling each instance equally from the mixed datasets. Furthermore, increasing the sampling ratio of high-quality collections (_e.g.,_ FLAN [67] and P3 [167]) can generally lead to performance improvement according to recent findings [95, 69]. Further, it is common to set a _maximum cap_ to control the maximum number of examples that a dataset can contain during instruction tuning [82], which is set to prevent larger datasets from overwhelming the entire distribution [82, 95]. In practice, the maximum cap is typically set to several thousands or tens of thousands according to different datasets [67, 69]. Recently, it has been empirically found that existing instruction datasets (Table III) mainly focus on enhancing LLMs' capabilities in certain aspects, and a single dataset alone cannot lead to a comprehensive enhancement in model capacity [353]. Therefore, it is often suggested to use a mixture of existing instruction datasets to achieve a balanced improvement in different capacities, including NLP task data (_e.g.,_ FLAN v2 [292]), chat data (_e.g.,_ ShareGPT [148]), and synthetic data (_e.g.,_ GPT4-Alpaca [354]).