Knowledge distillation is firstly introduced in [1]} and then brought back to popularity by [2]}.
The rationale behind is to use a student model (\(S\) ) to learn from a teacher model (\(T\) ) without sacrificing much accuracy compared to the teacher.
Challenges lie in two aspects, which are (1) how to extract the knowledge from \(T\) , and (2) how to transfer the knowledge to \(S\) .
