Recently, there are many new online distillation methods that train a teacher and a student simultaneously during knowledge transfer.
Collaborative learning is the one used most often [1]}, [2]}, [3]}, [4]}, [5]}, [6]},
where the teacher and the student as peer networks collaboratively teach and learn from each other, and the peer network architectures can be different. In particular, Zhang et al. [1]} proposed a deep mutual learning method (DML) for online distillation using the response-based knowledge. DML uses an ensemble of soft logits as knowledge and transfers it among arbitrary peer networks via collaborative learning [5]}. Yao and Sun [4]} further extended DML with dense cross-layer mutual-distillation, which learns both the teacher and the student collaboratively from scratch.
