Knowledge distillation.Knowledge distillation Hinton et al. (2015); Sanh et al. (2019); West et al. (2021); Magister et al. (2022) often involves the transfer of knowledge from larger models to smaller ones. Self-Instruct can also be viewed as a form of "knowledge distillation", however, it differs from this line in the following ways: (1) the source and target of distillation are the same, i.e., a model's knowledge is distilled to itself; (2) the content of distillation is in the form of an instruction task (i.e., instructions that define a task, and a set of examples that instantiate it).