Knowledge distillation usually utilizes a teacher-student strategy, which targets at transferring the knowledge from a strong teacher model to a student model [1]}, [2]}, [3]}. Previous purpose of knowledge distillation lies on model compression, which intends to reduce the large model space of a teacher model to a small student model while trying best to keep the performance. Techniques include soft label matching [3]}, adding random perturbations into soft label [5]}, hidden layer approximating [6]} and attention weight mapping [7]}.
