Language model pre-training, such as BERT, has significantly improved the performances of many natural language processing tasks. However, pre-trained language models are usually computationally expensive, so it is difficult to efficiently execute them on resource-restricted devices. To accelerate inference and reduce model size while maintaining accuracy, we first propose a novelTransformer distillationmethod that is specially designed for knowledge distillation (KD)  of the Transformer-based models. By leveraging this new KD method, the plenty of knowledge encoded in a large “teacher” BERT can be effectively transferred to a small “student” TinyBERT. Then, we introduce a newtwo-stage learningframework for TinyBERT, which performs Transformer distillation at both the pre-training and task-specific learning stages. This framework ensures that TinyBERT can capture the general-domain as well as the task-specific knowledge in BERT. TinyBERT4111The code and models are publicly available athttps://github.com/huawei-noah/Pretrained-Language-Model/tree/master/TinyBERTwith 4 layers is empirically effective and achieves more than 96.8% the performance of its teacher BERTBASEon GLUE benchmark, while being7.5x smallerand9.4x fasteron inference. TinyBERT4is also significantly better than 4-layer state-of-the-art baselines on BERT distillation, with only28%parameters and31%inference time of them. Moreover, TinyBERT6with 6 layers performson-par withits teacher BERTBASE.