It is commonly perceived that the strongest language models (LMs)  rely on a combination of massive scale, instruction data, and human feedback to perform specialized tasks –e.g.summarization and paraphrasing, without supervision. In this paper, we propose that language models can learn to summarize and paraphrase sentences, with none of these 3 factors. We presentImpossible Distillation, a framework that distills a task-specific dataset directly from an off-the-shelf LM, even when it is impossible for the LM itself to reliably solve the task. By training a student model on the generated dataset and amplifying its capability through self-distillation, our method yields ahigh-qualitymodel and dataset from alow-qualityteacher model, without the need for scale or supervision. UsingImpossible Distillation, we are able to distill an order of magnitude smaller model (with only 770M parameters)  that outperforms 175B parameter GPT-3, in both quality and controllability, as confirmed by automatic and human evaluations. Furthermore, as a useful byproduct of our approach, we obtainDimSum+, a high-quality dataset with 3.4M sentence summaries and paraphrases. Our analyses show that this dataset, as a purely LM-generated corpus, is more diverse and more effective for generalization to unseen domains than all human-authored datasets – including Gigaword with 4M samples.