Large language models have recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks. It has been hypothesized that this is a consequence of implicit multitask learning in language models’ pretraining. Can zero-shot generalization instead be directly induced byexplicitmultitask learning? To test this question at scale, we develop a system for easily mapping any natural language tasks into a human-readable prompted form.
We convert a large set of supervised datasets, each with multiple prompts with diverse wording.
These prompted datasets allow for benchmarking the ability of a model to perform completely held-out tasks.
We fine-tune a pretrained encoder-decoder modelon this multitask mixture covering a wide variety of tasks.
The model attains strong zero-shot performance on several standard datasets, often outperforming models up toits size.
Further, our approach attains strong performance on a subset of tasks from the BIG-bench benchmark, outperforming models up to 6its size.
All trained models are available athttps://github.com/bigscience-workshop/t-zero, and all prompts are available athttps://github.com/bigscience-workshop/promptsource.