In student-teacher model compression, it is desirable that the student networks should learn as much as possible from the teacher [1]}, with the goal of maximizing the mutual information between the teacher and student. For this reason, the student loss function often contains not simply a cross-entropy softmax for the labels, but also loss terms for the student to match the teacher's features (the features from the teacher's hidden layers are also called “dark information”). This includes deep supervision, in which features at several layers are matched and not just the final hidden layer's features [2]}, [3]}.
