Knowledge distillation offers the promise to boost the performance of such compact networks by exploiting deeper teacher architectures. Early work in this space focused on the task of image classification. In particular, [1]} proposed to distill the teacher's class probability distribution into the student, and [2]} encouraged the student's intermediate feature maps to mimic the teacher's ones. These initial works were followed by a rapid growth in the number of knowledge distillation strategies, including methods based on attention maps [3]}, on transferring feature flows defined by the inner product of features [4]}, and on contrastive learning to structure the knowledge distilled from teacher to the student [5]}. [6]} provides a comprehensive study of feature distillation.
