Pre-trained language models (e.g., BERTand its variants)  have achieved remarkable success in varieties of NLP tasks.
However, these models usually consist of hundreds of millions of parameters which brings challenges for fine-tuning and online serving in real-life applications
due to latency and capacity constraints.
In this work, we present a simple and effective approach to compress large Transformerbased pre-trained models, termed as deep self-attention distillation.
The small model (student)  is trained by deeply mimicking the self-attention module, which plays a vital role in Transformer networks, of the large model (teacher) .
Specifically, we propose distilling the self-attention module of the last Transformer layer of the teacher, which is effective and flexible for the student.
Furthermore, we introduce the scaled dot-product between values in the self-attention module as the new deep self-attention knowledge, in addition to the attention distributions (i.e., the scaled dot-product of queries and keys)  that have been used in existing works.
Moreover, we show that introducing a teacher assistantalso helps the distillation of large pre-trained Transformer models.
Experimental results demonstrate that our monolingual model111The code and models are publicly available athttps://aka.ms/minilm.outperforms state-of-the-art baselines in different parameter size of student models.
In particular, it retains more thanaccuracy on SQuAD 2.0 and several GLUE benchmark tasks usingof the Transformer parameters and computations of the teacher model.
We also obtain competitive results in applying deep self-attention distillation to multilingual pre-trained models.