We conjecture that the above “overall improvement” on the balanced test may cover up the fact that today's long-tailed classification methods are still biased and far from truly “improved”. Figure REF  (a.1) and (b.1) show an evidence. Regardless of the test distributions, balanced or imbalanced, an imbalanced model (vanilla cross-entropy trained, XE, orange line) always predicts the class distribution shaped as its own model prior, which is long-tailed. As a comparison, a balanced model (a SOTA method TDE [1]}, blue line) always predicts the distribution to cater the tail. This observation implies that both of them are biased: the imbalanced model is “head” biased; the balanced one is “tail” biased, which merely replaces the “head” bias with the “tail” bias.
Therefore, we expect an “unbiased” model's prediction to treat each class fairly and respects both test distributions, i.e., the prediction distribution follows the shape of the ground-truth one (e.g., green line).
Figure REF  (a.2) and (b.2) provide another evidence based on the true positive (TP) and the false positive (FP) statistics. We observe that the balanced model TDE tends to predict more samples as tail classes. On the contrary, the imbalanced model XE prefers to predict more samples as head classes. The preference on head or tail classes comes with a large number of FP samples, which lowers the precision score. In contrast, our xERM has no significant preference between head and tail classes, which achieves high precision under both test distributions.
Only through the above unbiasedness, we can diagnose whether a model truly improves long-tailed classification,
but not by playing a “prior flipping” game (cf. Section REF ).
<FIGURE>