Analysis of weights. Recall that the weights \(w^\text{imba}\)  and \(w^\text{ba}\)  control the balance between two cross-entropy losses. Some readers may wonder whether the improvement comes from knowledge distillation, which shares similar formulation except for using constant weights \(w \in [0,1]\)  and \(1-w\)  for \(w^\text{ba}\)  and \(w^\text{imba}\)  to blend the KL divergence loss and cross-entropy loss. In particular, \(w\!=\!0\)  denotes that model is only trained using the XE loss while \(w\!=\!1\)  denotes self-distillation [1]}. Figure REF  (a) illustrates the results of using constant weights ranging from 0 to 1. As shown in Figure REF  (a), knowledge distillation indeed can improve the performance compared to XE. However, it exhibits a large performance gap with xERM. This indicates that xERM is not a simple knowledge distillation approach.
