However, a powerful teacher is not always available. The recently proposed online knowledge distillation discards pre-trained teacher model, and trains multiple student models by aggregating and aligning their outputs [1]}, [2]}, [3]}. Deep mutual learning [2]} trained a set of student models collaboratively by learning from each other throughout the training process, which also increases the model robustness. OKDDip [3]} performed two-level distillation by introducing multiple auxiliary peers and one team leader, which achieved good performance without increasing computational complexity in inference.
