Knowledge distillation (KD) [1]} is a method of transferring learning from one model to the other, usually by compression, where a larger teacher model supervises the training of a smaller student model. One of the benefits of KD is that it can handle heterogeneous structures, i.e., the teacher and the student do not need to share the same network structure. Instead, the teacher supervises the student training via its logits, also called the soft target. In other words, KD minimizes the distance between the student network output \(\hat{z}\)  and the logits \(z\)  from a teacher network, generated from an arbitrary input sample:
\(\mathcal {L}_{KD} = ||\hat{z} - z||_2^2\) 
