Knowledge distillation (KD) was first proposed by Hinton et al. [1]}, also known as the teacher-student framework.
The goal of the KD mechanism is to transfer the knowledge from a complex model (Teacher) to a simple one (Student), which is often used to compress model size.
Recently, many different KD strategies have been proposed, including output transfer [1]}, [3]}, feature transfer [4]}, [5]}, and relation transfer [6]}.
All these mechanisms have been widely used in the image classification task and achieved remarkable results.
Meanwhile, some researchers are also trying to introduce the KD mechanism into the image restoration tasks.
For example, Hong et al. [7]} proposed a KD-based method for image dehazing with the heterogeneous task.
Specifically, the teacher model is an off-the-shelf autoencoder network, used to learn the reconstruction process from clear to clear images and transfer the learned knowledge to the dehazing model during the training process.
Learning knowledge from clear images is an interesting idea.
However, our investigation shows that the reconstruction of clear to clear images using a flat network such as U-net [8]}, is a relatively simple task that is difficult to ensure all layers or modules in the teacher model are fully trained since a flat network with skip connection could directly transmit all original information to the last layer. Therefore, it will make the teacher model fail to guide the student model efficiently.
In this paper, we aim to explore more interpretable knowledge distillation strategies for SID to further improve the denoising ability of the tiny model, especially under high noise levels.
