Transferring different types of knowledge can be implemented with different distillation methods, e.g., offline distillation, online distillation and self-distillation [1]}. Most of the KD methods employ offline distillation, which is one-way knowledge transfer from a pre-trained large teacher to a small student [2]}, [3]}, [4]}. In offline distillation, the capacity gap caused by a fixed teacher-student architecture and the requirement of a large dataset for pre-training the teacher often
result in a degraded performance [3]}. Thus, finding a proper teacher-student architecture in offline distillation
is challenging. In contrast, online distillation provides a one-phase end-to-end training scheme via teacher-student collaborative learning on a peer-network architecture instead of a fixed one [6]}, [7]}, [8]}, [9]}, [10]}, [11]}. Self-distillation performs online distillation within the same network to reduce model over-fitting [12]}, [13]}. Online distillation and self-distillation are promising methods for knowledge distillation as they bridge
the capacity gap via avoiding the need of a large teacher network, leading to an improved performance.
However, both KD methods used individually are limited to knowledge distillation from a single source, i.e., individual instances,
online distillation could further suffer from the poor instance consistency between peer networks caused by the discrepancy in their network outputs.
<FIGURE>