Humans (e.g., crowdworkers)  have a remarkable ability in solving different tasks, by simply reading textualinstructionsthat define them and looking at a few examples.
Despite the success of the conventional supervised learning on individual datasets, such models
often struggle with generalization across tasks (e.g., a question-answering system cannot solve classification tasks) .
A long-standing challenge in AI is to build a model that learns a new task by understanding the human-readableinstructionsthat define it.
To study this, we introduceNatural Instructions, a dataset of
61 distinct tasks, their human-authored instructions, and 193task instances (input-output pairs) . The instructions are obtained from crowdsourcing instructions used to create existing NLP datasets and mapped to a unified schema.
Using this meta-dataset, we measure cross-task generalization by training models onseentasks and measuring generalization to the remainingunseenones.
We adopt generative pre-trained language models to encode task-specific instructions along with input and generate task output.
Our results indicate that modelsbenefit from instructionswhen evaluated in terms of generalization to unseen tasks (19% better for models utilizing instructions) .
These models, however, are far behind
an estimated performance upperbound,
indicating significant room for more progress in this direction.111Dataset is available athttps://instructions.apps.allenai.org