Additionally, to compare Self-Instruct training with other publicly available instruction tuning data, we further finetune GPT3 model with data from PromptSource and SuperNI, which are used to train the T0 and T_k_-Instruct models. We call them T0 training and SuperNI training for short, respectively. To save the training budget, we sampled 50K instances (but covering all their instructions) for each dataset, which has a comparable size to the instruction data we generated. Based on the findings from Wang et al. (2022) and our early experiments, reducing the number of instances per training task does not degrade the model's generalization performance to unseen tasks.