We study the design decisions of publicly available instruction tuning methods, and break down the development of Flan 2022 models.
Through careful ablation studies on the Flan Collectionof instruction tuning tasks and methods, we tease apart the effect of design decisions that enable Flan-T5 to outperform prior work by 3-17%+ across evaluation settings.
We find task balancing and enrichment techniques are overlooked but critical to effective instruction tuning, and in particular, training with mixed prompt settings (zero-shot, few-shot, and chain-of-thought)  actually yields stronger (2%+)  performance inallsettings.
In further experiments, we show Flan-T5 requires less finetuning to converge higher and faster than T5 on single downstream tasks—motivating instruction-tuned models as more computationally-efficient starting checkpoints for new tasks.
Finally, to accelerate research on instruction tuning, we make the Flan 2022 collection of datasets, templates, and methods publicly available.111Data generation code available at:https://github.com/google-research/FLAN/tree/main/flan/v2. Generation code allows users to vary mixtures rates, templates, prompt types and data augmentations techniques, for faster public research.