Bootstrapping with limited resources.A series of recent works use language models to bootstrap some inferences using specialized methods. NPPPrompt Zhao et al. (2022) provides a method to generate predictions for semantic labels without any finetuning. It uses a model's own embeddings to automatically find words relevant to the label of the data sample and hence reduces the dependency on manual mapping from model prediction to label (verbalizers). STAR Zelikman et al. (2022) iteratively leverages a small number of rationale examples and a large dataset without rationales, to bootstrap a model's ability to perform reasoning. Self-Correction Welleck et al. (2023) decouples an imperfect base generator (model) from a separate corrector that learns to iteratively correct imperfect generations and demonstrates improvement over the base generator. Our work instead focuses on bootstrapping new tasks in the instruction paradigm.