Revisiting Self-Distillation
Knowledge distillation is the procedure of transferring "knowledge" from a
large model (the teacher) to a more compact one (the student), often being used
in the context of model compression. When both models have the same
architecture, this procedure is called self-distillation. Several works have
anecdotally shown that a self-distilled student can outperform the teacher on
held-out data. In this work, we systematically study self-distillation in a
number of settings. We first show that even with a highly accurate teacher,
self-distillation allows a student to surpass the teacher in all cases.
Secondly, we revisit existing theoretical explanations of (self) distillation
and identify contradicting examples, revealing possible drawbacks of these
explanations. Finally, we provide an alternative explanation for the dynamics
of self-distillation through the lens of loss landscape geometry. We conduct
extensive experiments to show that self-distillation leads to flatter minima,
thereby resulting in better generalization.