To reduce the model size while preserving accuracy, various model compression techniques have been proposed such as: pruning, quantization, and Knowledge Distillation (KD) [1]}. Among these methods, task-aware KD is a popular and particularly promising approach for compressing Transformer-based models [1]}. The general idea is to first fine-tune a large model (namely the teacher model) based on the task's labeled data, and then train a separate network that has significantly fewer parameters (namely the student model) than the original model to mimic the predictions of the original large model.
A large number of task-aware KD algorithms have been proposed in the NLP regime, e.g., DistillBERT [3]}, BERT-PKD [4]}, BERT-EMD [5]}, TinyBERT [6]}, DynaBERT [7]}, and AdaBERT [8]}, some of which can compress the teacher network by 10\(\times \)  without significant accuracy loss on certain datasets.
