CTLS-MKT unifies student self-learning and teacher-student mutual learning under one framework in such a way that it can utilise multiple types of knowledge and more than one distillation methods during the teacher-student learning. The teacher-student architectures used in CTSL-MKT are peer networks, such as ResNet [1]} and MobilenetV2 [2]}. Different from previous works, CTSL-MKT distills both the response-based knowledge from individual instance features and the relation-based knowledge from instance relationships. During teacher-student mutual learning, peer networks trained collaboratively can teach each other via online distillation with the two kinds of knowledge. Meanwhile, each peer network can also self-learn via self-distillation with the response-based knowledge. The two leaning processes working together can complement each other to explore different knowledge space and enhance the learning. Our CTSL-MKT can be seen as a new model compression technique that generalises the existing self-distillation and online distillation methods and
enables fast computations and improves the generalization ability.
The overall framework of CTSL-MKT with two peer networks is shown in Figure REF 
as an example, and notations are summarized in Table REF .
<TABLE>