In knowledge distillation, the type of knowledge, the distillation strategy and the teacher-student architecture are three crucial factors that determine the KD performance [1]}. As pointed out in [1]}, there are three kinds of knowledge, i.e., the response-based, the feature-based and the relation-based knowledge. Generally, most of KD methods distill the response-based knowledge (e.g., soft logits of the output layer) from a large teacher network and transfer it to a small student [3]}, [4]}, [5]}. To overcome the limitation of knowledge from the output layer of teacher, the feature-based knowledge from the middle layers of teacher is
also used to train the student [6]}, [7]}, [8]}. Unlike both the response-based and the feature-based knowledge from individual instances, the relation-based knowledge from instance relations is modelled for improving student learning [9]}, [10]}, [11]}, [12]}, [13]}.
Each kind of knowledge can provide student training with an informative teacher guidance, and they can also compensate each other to enrich learning.
However, most existing KD methods only consider either knowledge from individual instance features to maintain instance consistency between teacher and student or knowledge from instance relations to preserve the instance correlation consistency.
There are a few works that consider more than one kind of knowledge in knowledge distillation [9]}, [15]} at the same time and explore the efficacy of each kind of knowledge.
