before by NLP practitioners (though potentially still important for real users). In parallel with our work, Honovich et al. (2022) also propose to generate large-scale instruction data (so-called Unnatural Instructions) with GPT3 models. The major differences are that 1) they use tasks in SuperNI Wang et al. (2022) as their seed tasks, resulting in a different distribution of generated tasks; 2) they employ InstructGPT\({}_{002}\) for generating the data, in which sense they are distilling knowledge from an already instruction-tuned model, while we solely rely on the vanilla LM; 3) the detailed generation pipeline and templates are different. Nevertheless, we believe that both efforts in expanding instruction data are complementary, and the community will benefit from these diverse datasets.