XAI methods provide complementary, post-hoc explanations to the predictions of black-box models to induce trust. They are usually sensitivity analyses, producing systematic perturbations of the inputs to see how predictions are affected [1]}. They can either provide (i) feature importance explanations, that highlight top features responsible for a prediction, like LIME [2]}, SHAP [3]}, DeepLift [4]}, or (ii) example importance explanations, that highlight top training examples most responsible for a prediction, like Deep K-Nearest Neighbours [5]}. Uncertainty estimation is a rich field with a long history.
Classical techniques like density estimation [6]}, one-class SVMs [7]}, tree-isolation forests [8]}, etc.,
scale badly with input dimensionality [9]}.
[10]} provides a “trust" score using level-set estimation and modified nearest neighbors search. However, it does not scale well to high dimensional datasets.
Calibration is a frequentist notion of uncertainty [11]}, [12]}, [13]}, measured by proper scoring rules like log-loss or Brier scoring. Deep neural networks (NNs) typically use a Bayesian formalism to learn distributions over model weights [14]}, [15]}, [16]}, [17]}, [18]}, [19]} or approximate Bayesian inference such as Monte-Carlo Dropout [20]} and Batch Normalization [21]}. Reconstruction-based methods [22]}, [23]}, [24]}, [25]}, [26]} use reconstruction loss as the uncertainty score.
Quality of uncertainty estimates are commonly evaluated via OOD detection, a binary classification task.
[27]} uses maximum softmax probabilites (MSP) to detect OOD data. [28]} introduces a temperature parameter to the softmax equation. [29]} fits class-conditional Gaussians to intermediate activations and uses the Mahalanobis distance to identify OOD samples. Self-supervision helps get better representations that improves OOD detection [30]}, [31]}, [32]}. Some consider (semi-)supervised techniques for OOD detection with labelled outliers [33]}, [34]}, [35]}, [36]}, [37]}, [38]}. Others use
generative techniques [39]}, [40]}, [41]} including energy-based models [42]}, [43]}, [44]} and likelihood ratios [45]},
but they can be overconfident on complex inputs [46]}. Though many directly optimize for good performance on OOD detection,
few are compatible to
serve as trust scoring frameworks for continuous model monitoring.
Our sequential
mistrust scores build upon and shares similarity to change point detection (CP) methods in time-series data [47]}, [48]}, along with signal processing techniques such as Particle and Kalman filtering [49]}, [50]}. However, traditional CP methods are limited to one (or low) dimensional data and fail in high dimensional settings [51]}.
