Knowledge distillation refers to a technique that transfers the representation ability of a large (Teacher) model to a small one (Student) for enhancing the performance of the student model. Hence, it has been widely used for network compression or to further improve the performance of the student model, which has shown the effectiveness in many computer vision tasks. Meanwhile, there are mainly two kinds of knowledge distillation, soft label distillation and feature distillation. In soft label distillation, the softmax outputs of a teacher model are regarded as soft labels to provide informative dark knowledge to the student model [1]}. In feature distillation, the intermediate features maps are transferred to the student model [2]}, [3]}.
