Deep neural networks (DNNs) have achieved great success in various computer vision [1]}, natural language processing [2]} and speech recognition [3]} tasks. However, those well-performed DNNs typically require very high computation and large memory usage, making them difficult to be deployed on platforms with limited resources.
To solve this problem, knowledge distillation was proposed as one of the mainstream model compression methods [4]}, [5]}, [6]}, [7]}, [8]}, [9]}, [10]}, [11]}. Knowledge distillation (KD) aims to transfer knowledge from a pre-trained, cumbersome and powerful teacher model to a compressed student model with fewer parameters, hoping that the student model can achieve better performance than training with labels alone and even be on par with the teacher model.
<FIGURE>