How well can NLP models generalize to avarietyof unseen tasks when provided with task instructions?
To address this question, we first introduceSuper-NaturalInstructions,111Super-NaturalInstructionsrepresents a super-sized expansion ofNaturalInstructionswhich had 61 tasks.a benchmark of 1,616 diverse NLP tasks and their expert-written instructions.
Our collection covers 76 distinct task types,
including but not limited to classification, extraction, infilling, sequence tagging, text rewriting, and text composition.
This large and diverse collection of tasks enables rigorous benchmarking of cross-task generalization under instructions—training models to follow instructions on a subset of tasks and evaluating them on the remaining unseen ones.Furthermore, we buildT-Instruct, a transformer model
trained to follow a variety of in-context instructions (plain language task definitions or-shot examples) .
Our experiments show thatT-Instructoutperforms existing
instruction-following models such as InstructGPT by over 9% on our benchmark despite being an order of magnitude smaller.
We further analyze generalization as a function of various scaling parameters, such as the number of observed tasks, the number of instances per task, and model sizes.
We hope our dataset and model facilitate future progress towards more general-purpose
NLP models.222The dataset, models, and
a leaderboard
can be found athttps://instructions.apps.allenai.org.††Co-first authorsCo-second authors