Recent advancements in Natural Language Processing (NLP) such as BERT [1]}, RoBERTa [2]}, and ELECTRA [3]} have demonstrated the effectiveness of Transformer models in generating transferable language representations.
Pretraining over a massive unlabeled corpus and then fine-tuning over labeled data for the task of interest has become the state-of-the-art paradigm for solving diverse NLP problems ranging from sentence classification to question answering [4]}.
Scaling up the size of these networks has led to rapid NLP improvements. However, these improvements have come at the expense of significant increase of memory for their many parameters and compute to produce predictions [5]}, [6]}.
This prevents these models from being deployed on resource-constrained devices such as smart phones and browsers, or for latency-constrained applications such as click-through-rate prediction. There is great demand for models that are smaller in size, yet still retain similar accuracy as those having a large number of parameters.
