Most of knowledge distillation approaches adopt the output logits of a deep model from individual samples as knowledge and make the logits of the teacher and student match each other. However, such response-based knowledge ignores the structural knowledge from the mutual relations of data examples, known as relation-based knowledge. In recent years, there are some newly proposed knowledge distillation methods based on structural relations of data samples [1]}, [2]}, [3]}, [4]}, [5]}, [6]}. Park et al. [1]} proposed a relational knowledge distillation method (RKD), which transfers the instance relation knowledge from a teacher to a student. Chen et al. [6]} borrowed the idea of manifold learning to design a novel knowledge distillation method, in which the student preserves the feature embedding similarities of samples from the teacher. Peng et al. [3]} designed a knowledge transfer method that makes sure the student matches the instance correlation consistently with the teacher.
