Finetuned Language Models Are Zero Shot Learners
This paper explores a simple method for improving the zero shot learning abilities of language models. We show that instruction tuning finetuning language models on a collection of datasets described via instructions substantially boosts zeroshot performance on unseen tasks. We take a 137B parameter pretrained language model and instruction tune it on over 60 NLP datasets verbalized via natural language instruction templates. We evaluate this instruction tuned model, which we call FLAN, on unseen task types. FLAN substantially improves the performance of its unmodified counterpart and surpasses zero shot 175B GPT 3 on 20 of 25 datasets that we evaluate. FLAN even outperforms few shot GPT 3 by a large margin on ANLI, RTE, BoolQ, AI2 ARC, OpenbookQA, and StoryCloze. Ablation studies reveal that number of datasets and model scale are key components to the success of instruction tuning. Target Input (Commonsense Reasoning) keep stack of pillow cases in fridge Inference on unseen task type Finetune on many tasks "instruction tuning" Translate this sentence to Spanish: The new office building was built in less than three months. Input (Translation) El nuevo edificio de oficinas se construyo en tres meses. Target Input (Natural Language Inference) It is not possible to tell FLAN Response Coreference resolution tasks Sentiment analysis tasks GPT 3 175B zero shot GPT 3 175B few shot FLAN 137B zero shot Performance on unseen task types Natural language inference 42.9 53.2 56.2 Reading Comprehension 63.7 72.6 77.4 Closed Book QA 49.8 55.7 56.6 Here is a goal: Get a cool sleep on summer days. How would you accomplish this goal? OPTIONS: Keep stack of pillow cases in fridge. Keep stack of pillow cases in oven. Premise: At my age you will probably have learnt one lesson. Hypothesis: It's not certain how many lessons you'll learn by your thirties. Does the premise entail the hypothesis? OPTIONS: yes it is not possible to tell no Figure 1: Top: overview of instruction tuning and FLAN. Instruction tuning finetunes a pretrained language model on a mixture of tasks phrased as instructions. At inference time, we evaluate on an unseen task type; for instance, we could evaluate the model on natural language inference (NLI) when no NLI tasks were seen during instruction tuning. Bottom: performance of zero shot FLAN on unseen task types, compared with zero shot GPT 3 and few shot. NLI datasets: ANLI R1 R3, CB, RTE. Reading comprehension datasets: BoolQ, MultiRC, OBQA. Closed book QA datasets: ARC easy, ARC challenge, NQ, TriviaQA. *Lead contributors. Author contributions listed at end of paper. 1 ar X iv :2 10 9. 01 65 2v 2 cs .C L 5 O ct 2 02 1
ArXiv