Crosslingual Generalization through Multitask Finetuning
Niklas Muennighoff1Thomas Wang1Lintang Sutawika2,3Adam Roberts4Stella B
Crosslingual Generalization through Multitask Finetuning
Multitask prompted finetuning (MTF) has been shown to help large language
models generalize to new tasks in a zero-shot setting, but so far explorations
of MTF have focused on English data and models. We apply MTF to the pretrained
multilingual BLOOM and mT5 model families to produce finetuned variants called
BLOOMZ and mT0. We find finetuning large multilingual language models on
English tasks with English prompts allows for task generalization to
non-English languages that appear only in the pretraining corpus. Finetuning on
multilingual tasks with English prompts further improves performance on English
and non-English tasks leading to various state-of-the-art zero-shot results. We
also investigate finetuning on multilingual tasks with prompts that have been
machine-translated from English to match the language of each dataset. We find
training on these machine-translated prompts leads to better performance on
human-written prompts in the respective languages. Surprisingly, we find models
are capable of zero-shot generalization to tasks in languages they have never
intentionally seen. We conjecture that the models are learning higher-level
capabilities that are both task- and language-agnostic. In addition, we
introduce xP3, a composite of supervised datasets in 46 languages with English
and machine-translated prompts. Our code, datasets and models are freely
available at https://github.com/bigscience-workshop/xmtf.
Niklas Muennighoff Thomas Wang Lintang Sutawika Adam Roberts Stella Biderman Teven Le Scao M Saiful Bari Sheng Shen Zheng-Xin Yong Hailey Schoelkopf Xiangru Tang Dragomir Radev Alham Fikri Aji Khalid Almubarak Samuel Albanie Zaid Alyafeai Albert Webson Edward Raff Colin Raffel