UL2: Unifying Language Learning Paradigms
Yi Tay∗Mostafa Dehghani
Vinh Q. Tran]Xavier Garcia]Jason Wei]Xuezhi Wang]Hyung Won Ch
UL2: Unifying Language Learning Paradigms
Existing pre-trained models are generally geared towards a particular class
of problems. To date, there seems to be still no consensus on what the right
architecture and pre-training setup should be. This paper presents a unified
framework for pre-training models that are universally effective across
datasets and setups. We begin by disentangling architectural archetypes with
pre-training objectives -- two concepts that are commonly conflated. Next, we
present a generalized & unified perspective for self-supervision in NLP and
show how different pre-training objectives can be cast as one another and how
interpolating between different objectives can be effective. We then propose
Mixture-of-Denoisers (MoD), a pre-training objective that combines diverse
pre-training paradigms together. We furthermore introduce a notion of mode
switching, wherein downstream fine-tuning is associated with specific
pre-training schemes. We conduct extensive ablative experiments to compare
multiple pre-training objectives and find that our method pushes the
Pareto-frontier by outperforming T5 & GPT-like models across multiple diverse
setups. By scaling our model up to 20B parameters, we achieve SOTA performance
on 50 well-established supervised finetuning based NLP tasks. Our model also
achieve strong results at in-context learning, outperforming 175B GPT-3 on
zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot
summarization. On 0-shot MMLU, UL2 20B outperforms T0 and T5 models. UL2 20B
also works well with chain-of-thought prompting and reasoning, making it an
appealing choice for research into reasoning at a small to medium scale of 20B
parameters. Finally, we apply FLAN instruction tuning to the UL2 20B model,
achieving MMLU and Big-Bench scores competitive to FLAN-PaLM 62B. We release
Flax-based T5X checkpoints for the UL2 20B & Flan-UL2 20B.
Yi Tay Mostafa Dehghani Vinh Q. Tran Xavier Garcia Jason Wei Xuezhi Wang Hyung Won Chung Siamak Shakeri Dara Bahri Tal Schuster Huaixiu Steven Zheng Denny Zhou Neil Houlsby Donald Metzler