Knowledge distillation was first proposed by [1]}, which used soft targets to transfer knowledge from the teacher model to the student model. The soft targets, i.e., the teacher model predictions can capture the relationship among classes. In addition to the output layer, there is also a lot of works to extract knowledge from the intermediate layers [2]}, [3]}, [4]}, [5]}. FitNet [2]} added an additional fully-connected layer to the intermediate student output and forced it to match the intermediate feature maps of the teacher model. Attention transfer [3]} extracted attention maps instead of features. SemCKD [4]} used an attention mechanism to automatically assign the appropriate target layer of the teacher model to each student layer.
