Despite the recent success of multi-task learning and transfer learning for natural language processing (NLP) , few works have systematically studied the effect of scaling up the number of tasks during pre-training. Towards this goal, this paper introducesExMix(ExtremeMixture) : a massive collection of 107 supervised NLP tasks across diverse domains and task-families. UsingExMix, we study the effect of multi-task pre-training at the largest scale to date, and analyze co-training transfer amongst common families of tasks. Through this analysis, we show that manually curating an ideal set of tasks for multi-task pre-training is not straightforward, and that multi-task scaling can vastly improve models on its own. Finally, we proposeExT5: a model pre-trained using a multi-task objective of self-supervised span denoising and supervisedExMix. Via extensive experiments, we show thatExT5outperforms strong T5 baselines on SuperGLUE, GEM, Rainbow, Closed-Book QA tasks, and several tasks outside ofExMix.ExT5also significantly improves sample efficiency while pre-training.