Instruction tuning aims to train large language models to generate responses that align with input instructions, thereby enable LLMs with conversational and task execution capabilities. Compared with standard LLM, SFT allows the behavior of the model to be more controllable and predictable, thereby achieving the purpose of aligning human will. Methods to build instruction tuning datasets include: (1) Pure manual annotation Conover et al. (2023). This method completely constructs instructions and answers manually, which is very time-consuming and laborious; (2) Converted from existing datasets Mishra et al. (2022); Sanh et al. (2022); Chung et al. (2022). Some studies use supervised data sets from NLP tasks to construct instruction tuning data; (3) Automatically generated using LLM Honovich et al. (2022); Wang et al. (2023); Xu et al. (2023); Ji et al. (2023); Xu et al. (2023). Others use existing LLMs to generate instruction tuning data. A common practice is to first manually annotate high-quality seed datasets, and then use LLM to expand the seed instructions and corresponding outputs, which can generate large-scale instruction tuning data with very little human annotation. However, the quality cannot be guaranteed and there is a certain amount of noisy data, which can lead to hallucinations.