Measuring Class-Imbalance Sensitivity of Deterministic Performance Evaluation Metrics
The class-imbalance issue is intrinsic to many real-world machine learning
tasks, particularly to the rare-event classification problems. Although the
impact and treatment of imbalanced data is widely known, the magnitude of a
metric's sensitivity to class imbalance has attracted little attention. As a
result, often the sensitive metrics are dismissed while their sensitivity may
only be marginal. In this paper, we introduce an intuitive evaluation framework
that quantifies metrics' sensitivity to the class imbalance. Moreover, we
reveal an interesting fact that there is a logarithmic behavior in metrics'
sensitivity meaning that the higher imbalance ratios are associated with the
lower sensitivity of metrics. Our framework builds an intuitive understanding
of the class-imbalance impact on metrics. We believe this can help avoid many
common mistakes, specially the less-emphasized and incorrect assumption that
all metrics' quantities are comparable under different class-imbalance ratios.