Model self-training.A typical self-training framework He et al. (2019); Xie et al. (2020); Du et al. (2021); Amini et al. (2022); Huang et al. (2022) uses trained models to assign labels to unlabeled data and then leverages the newly labeled data to improve the model. In a similar line, Zhou et al. (2022) use multiple prompts to specify a single task and propose to regularize via prompt consistency, encouraging consistent predictions over the prompts. This allows either finetuning the model with extra unlabeled training data, or direct application at inference time. While Self-Instruct has similarities with the self-training literature, most self-training methods assume a specific _target task_ as well as _unlabeled examples_ under it; in contrast, Self-Instruct produces a variety of tasks from scratch.