SLOVA: Uncertainty Estimation Using Single Label One-Vs-All Classifier
Deep neural networks present impressive performance, yet they cannot reliably
estimate their predictive confidence, limiting their applicability in high-risk
domains. We show that applying a multi-label one-vs-all loss reveals
classification ambiguity and reduces model overconfidence. The introduced SLOVA
(Single Label One-Vs-All) model redefines typical one-vs-all predictive
probabilities to a single label situation, where only one class is the correct
answer. The proposed classifier is confident only if a single class has a high
probability and other probabilities are negligible. Unlike the typical softmax
function, SLOVA naturally detects out-of-distribution samples if the
probabilities of all other classes are small. The model is additionally
fine-tuned with exponential calibration, which allows us to precisely align the
confidence score with model accuracy. We verify our approach on three tasks.
First, we demonstrate that SLOVA is competitive with the state-of-the-art on
in-distribution calibration. Second, the performance of SLOVA is robust under
dataset shifts. Finally, our approach performs extremely well in the detection
of out-of-distribution samples. Consequently, SLOVA is a tool that can be used
in various applications where uncertainty modeling is required.