It can be challenging to train multi-task neural networks that outperform or even match their single-task counterparts. To help address this, we propose using knowledge distillation where single-task models teach a multi-task model. We enhance this training with teacher annealing, a novel method that gradually transitions the model from distillation to supervised learning, helping the multi-task model surpass its single-task teachers. We evaluate our approach by multi-task fine-tuning BERT on the GLUE benchmark. Our method consistently improves over standard single-task and multi-task training. Building a single model that jointly learns to perform many tasks effectively has been a longstanding challenge in Natural Language Processing (NLP). However, applying multi-task NLP remains difficult for many applications, with multitask models often performing worse than their single-task counterparts BID30 BID1 BID25 . Motivated by these results, we propose a way of applying knowledge distillation BID3 BID0 BID14 so that single-task models effectively teach a multi-task model.Knowledge distillation transfers knowledge from a "teacher" model to a "student" model by training the student to imitate the teacher's outputs. In "born-again networks" BID10 , the teacher and student have the same neural architecture and model size, but surprisingly the student is able to surpass the teacher's accuracy. Intuitively, distillation is effective because the teacher's output distribution over classes provides more training signal than a one-hot label; BID14 suggest that teacher outputs contain "dark knowledge" capturing additional information about training examples. Our work extends born-again networks to the multi-task setting. We compare Single→Multi 1 born-again distillation with several other variants (Single→Single and Multi→Multi), and also explore performing multiple rounds of distillation (Single→Multi→Single→Multi) . Furthermore, we propose a simple teacher annealing method that helps the student model outperform its teachers. Teacher annealing gradually transitions the student from learning from the teacher to learning from the gold labels. This method ensures the student gets a rich training signal early in training but is not limited to only imitating the teacher.Our experiments build upon recent success in self-supervised pre-training BID7 BID28 and multi-task fine-tune BERT BID8 to perform the tasks from the GLUE natural language understanding benchmark BID41 . Our training method, which we call Born-Again Multi-tasking (BAM) 2 , consistently outperforms standard single-task and multi-task training. Further analysis shows the multi-task models benefit from both better regu- 1 We use Single→Multi to indicate distilling single-task "teacher" models into a multi-task "student" model. 2 Code is available at https://github.com/ google-research/google-research/tree/ master/bam larization and transfer between related tasks. We have shown that Single→Multi distillation combined with teacher annealing produces results consistently better than standard single-task or multi-task training. Achieving robust multi-task gains across many tasks has remained elusive in previous research, so we hope our work will make multi-task learning more broadly useful within NLP. However, with the exception of closely related tasks with small datasets (e.g., MNLI helping RTE), the overall size of the gains from our multi-task method are small compared to the gains provided by transfer learning from self-supervised tasks (i.e., BERT). It remains to be fully understood to what extent "self-supervised pre-training is all you need" and where transfer/multi-task learning from supervised tasks can provide the most value.
distilling single-task models into a multi-task model improves natural language understanding performance
{"src": "scitldr-AIC_train", "paper_id": "SylnYlqKw4"}