Here \(m_{i, j} \in [0, 1]\)  represents the layer mapping weight between the \(i\) -th teacher layer and \(j\) -th student layer, \(H_i^\text{T}, H_j^\text{S}\)  are the \(i\) -th and the \(j\) -th hidden states of the teacher and the student (i.e. their intermediate representations at layers \(i\)  and \(j\) ), \(\beta _1\) , \(\beta _2\)  control the strength of distilling from class probabilities produced by the teacher, and \(\gamma _1\)  and \(\gamma _2\)  control the strength of learning from ground truth data (\(x,y\) ) and synthesized (augmented) data (\(\hat{x},\hat{y}\) ). In Appendix, we illustrated how previous model distillation algorithms [1]}, [2]}, [3]} can be encompassed in the Distiller framework.
<FIGURE>