Auditing Visualizations: Transparency Methods Struggle to Detect Anomalous Behavior
Transparency methods such as model visualizations provide information that
outputs alone might miss, since they describe the internals of neural networks.
But can we trust that model explanations reflect model behavior? For instance,
can they diagnose abnormal behavior such as backdoors or shape bias? To
evaluate model explanations, we define a model as anomalous if it differs from
a reference set of normal models, and we test whether transparency methods
assign different explanations to anomalous and normal models. We find that
while existing methods can detect stark anomalies such as shape bias or
adversarial training, they struggle to identify more subtle anomalies such as
models trained on incomplete data. Moreover, they generally fail to distinguish
the inputs that induce anomalous behavior, e.g. images containing a backdoor
trigger. These results reveal new blind spots in existing model explanations,
pointing to the need for further method development.