While the concept of trust can be nuanced and context-driven [1]}, a (mis)trust scoring framework for continuous model monitoring must satisfy the following desiderata– it must be: post-hoc - use the trained, deployed model; explainable - help us understand why a prediction should or should not be trusted; and actionable - allow us to take an automated, concrete action for every prediction, e.g. accept or reject prediction, flag, alert a human, relabel data point or adaptively retrain the model. Most importantly, the framework must perform well to avoid algorithm aversion [2]}. It must (1) assign high “mistrust" scores when the model is faced with large distributional shifts (out-of-distribution (OOD) data) such as noisy, corrupt data or new semantic content (semantic OOD data), and (2) assign low “mistrust" scores on learnt distributions yet unseen data (in-distribution or InD data), allowing for generalization.
