We compare several recent methods for qualifying uncertainty or OOD detection in text categorization.
(i) MSP refers to maximum softmax probability, a baseline work of OOD detection [1]}.
(ii) DP refers to Monte Carlo Dropout [2]}, which applies dropout at train and test time. We run ten it times and use the average MSP as the uncertainty score.
(iii) TS is a post-hoc calibration method by temperature scaling [3]}. We fine-tune the temperature parameter via the validation set.
(iv) MRC denotes Manifold Regularization Calibration [3]}, which adopts on- and off-manifold regularization to improve the calibration of BERT.
(v) OE refers to Outlier Exposure [5]} that enforces uniform confidence on an auxiliary OOD dataset.
(vi) ENN [6]} is our base classifier, which uses deep learning models to explicitly model SL uncertainty.
Most of the baselines with softmax function use the negative of maximum softmax scores (\(-max_c fc(x)\) ) as the uncertainty score, which is similar to predictive entropy. ENN uses predictive entropy. Our proposed model uses vacuity as the detection score.
