Recent KD research in the domain of NLP has investigated how to efficiently transfer knowledge from pretrained Transformer models. [1]} proposed BERT-PKD that transfers the knowledge from both the final layer and intermediate layers of the teacher network. [2]} proposed the TinyBERT model that first distills the general knowledge of the teacher by minimizing the Masked Language Model (MLM) objective [3]}, with subsequent task-specific distillation. [4]} proposed a many-to-many layer mapping function leveraging the Earth Mover’s Distance to transfer intermediate knowledge.
Our paper differs from the existing work in that we provide a systematic analysis of the different components in single-stage task-aware KD algorithms in NLP and propose the first automated KD algorithm in this area.
