Utilizing Class Separation Distance for the Evaluation of Corruption Robustness of Machine Learning Classifiers
Robustness is a fundamental pillar of Machine Learning (ML) classifiers,
substantially determining their reliability. Methods for assessing classifier
robustness are therefore essential. In this work, we address the challenge of
evaluating corruption robustness in a way that allows comparability and
interpretability on a given dataset. We propose a test data augmentation method
that uses a robustness distance $\epsilon$ derived from the datasets minimal
class separation distance. The resulting MSCR (mean statistical corruption
robustness) metric allows a dataset-specific comparison of different
classifiers with respect to their corruption robustness. The MSCR value is
interpretable, as it represents the classifiers avoidable loss of accuracy due
to statistical corruptions. On 2D and image data, we show that the metric
reflects different levels of classifier robustness. Furthermore, we observe
unexpected optima in classifiers robust accuracy through training and testing
classifiers with different levels of noise. While researchers have frequently
reported on a significant tradeoff on accuracy when training robust models, we
strengthen the view that a tradeoff between accuracy and corruption robustness
is not inherent. Our results indicate that robustness training through simple
data augmentation can already slightly improve accuracy.