Existing methods have designed various types of knowledge to improve their performance.
Methods in [1]} treated the hard label predicted by \(T\)  as the underlying knowledge, with the assumption that the well-trained \(T\)  has already eliminated some label errors contained in the groundtruth data.
Hinton  [2]} argued that the soft label produced by \(T\) , the classification probabilities, can provide richer information.
Some work [3]}, [4]}, [5]} extracted the knowledge from \(T\)  by processing the hidden feature map.
AT [4]} averaged the feature map across channel dimension to obtain spatial attention map, while work in [5]} defined inter-layer flow by computing the inner product of two feature maps, and Lee  [8]} improved this idea with singular value decomposition (SVD). Jin  [9]} claimed that the representation of a converged heavy model is not easy to learn. As a result, an approach called route constrained optimization (RCO) was proposed, from the perspective of curriculum learning.
