Natural Language Processing (NLP)  has recently achieved great success by using huge pre-trained models with hundreds of millions of parameters. However, these models suffer from heavy model sizes and high latency such that they cannot be deployed to resource-limited mobile devices. In this paper, we propose MobileBERT for compressing and accelerating the popular BERT model. Like the original BERT, MobileBERT is task-agnostic, that is, it can be generically applied to various downstream NLP tasks via simple fine-tuning. Basically, MobileBERT is a thin version of, while equipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks. To train MobileBERT, we first train a specially designed teacher model, an inverted-bottleneck incorporatedmodel. Then, we conduct knowledge transfer from this teacher to MobileBERT. Empirical studies show that MobileBERT is 4.3smaller and 5.5faster thanwhile achieving competitive results on well-known benchmarks.
On the natural language inference tasks of GLUE,MobileBERTachieves a GLUE score of(lower than) ,
and 62 ms latency on a Pixel 4 phone. On the SQuAD v1.1/v2.0 question answering task,MobileBERTachieves a dev F1 score of(higher than) .