We compare our proposed method with a number of state-of-the-art knowledge distillation methods including KD [1]}, FitNets [2]}, AT [3]}, AB [4]}, RKD [5]} and RCO [6]}.
For KD, we set \(T=4\)  and \(\lambda =16, 9\)  for CIFAR-100 and ImageNet, following [1]}.
For FitNets, AT, AB, RKD, and RCO, the loss weight is set to \(10^2, 10^3, 10, 10, 10^2\) , following [3]}[6]}[4]}.
