Significant advances have been made in Natural Language Processing (NLP) modelling since the beginning of 2018. The new approaches allow for accurate results, even when there is little labelled data, because these NLP models can benefit from training on both task-agnostic and task-specific unlabelled data. However, these advantages come with significant size and computational costs.

 This workshop paper outlines how our proposed convolutional student architecture, having been trained by a distillation process from a large-scale model, can achieve 300x inference speedup and 39x reduction in parameter count. In some cases, the student model performance surpasses its teacher on the studied tasks. The last year has seen several major advances in NLP modelling, stemming from previous innovations in embeddings BID0 [2] BID2 and attention models BID3 [5] BID5 that allow Language Models (LMs) to be trained on very large corpuses : For instance ELMo BID6 , OpenAI Transformer BID7 and recently BERT BID8 .In addition, the power of building on LM-enhanced contextualised embeddings, using a fine-tuning approach on task-specific unlabelled data BID9 , has shown huge benefits for downstream tasks (such as text classification) -especially in a typical industrial setting where labelled data is scarce.In order to make use of these advances, this work shows how a model distillation process BID10 can be used to train a novel 'student' CNN structure from a much larger 'teacher' Language Model. The teacher model can be fine-tuned on the specific task at hand, using both unlabelled data, and the (small number of) labelled training examples available. The student network can then be trained using both labelled and unlabelled data, in a process akin to pseudo-labelling BID11 [13].Our results show it is possible to achieve similar performance to (and surpass in some cases) large attention-based models with a novel, highly efficient student model with only convolutional layers. For text classifications, mastery may require both high-level concepts gleaned from language under standing and fine-grained textual features such as key phrases. Similar to the larval-adult form analogy made in BID10 , high-capacity models with task-agnostic pre-training may be well-suited for task mastery on small datasets (which are common in industry). On the other hand, convolutional student architectures may be more ideal for practical applications by taking advantage of massively parallel computation and a significantly reduced memory footprint.Our results suggest that the proposed BlendCNN architecture can efficiently achieve higher scores on text classification tasks due to the direct leveraging of hierarchical representations, which are learnable (even in a label-sparse setting) from a strong teaching model.Further development of specialized student architectures could similarly surpass teacher performance if appropriately designed to leverage the knowledge gained from a pretrained, task-agnostic teacher model whilst optimizing for task-specific constraints.
We train a small, efficient CNN with the same performance as the OpenAI Transformer on text classification tasks
{"src": "scitldr-AIC_train", "paper_id": "HJxM3hftiX"}