Self-distillation is a novel training scheme for knowledge transfer [1]}, [2]}, [3]}, [4]}, [5]}. In self-distillation, the teacher and student networks are identical and knowledge transfer is carried out within the same network. Yuan et al. empirically analyzed the performance of normal, reversed and defective KD methods, and showed that a weak teacher can strengthen the student and vice-versa [1]}. A teacher-free knowledge distillation method (Tf-KD) instead makes student model conduct self-learning. To enhance the generalization and overcome over-fitting, class-wise self-knowledge distillation makes use of soft logits of different intra-class samples within a model [3]}. Phuong and Lampert [4]} proposed a distillation-based training method to reduce time complexity, where the output of later exit layer supervises the early exit layer via knowledge transfer.
Rather than at the layer level, snapshot distillation [5]}
transfers knowledge from earlier to later epochs while training a deep model.
