@inproceedings{
raman2023do,
title={Do Concept Bottleneck Models Obey Locality?},
author={Naveen Raman and Mateo Espinosa Zarlenga and Juyeon Heo and Mateja Jamnik},
booktitle={XAI in Action: Past, Present, and Future Applications},
year={2023},
url={https://openreview.net/forum?id=F6RPYDUIZr}
}
Concept-based learning improves a deep learning model's interpretability by explaining its predictions via human-understandable concepts. Deep learning models trained under this paradigm heavily rely on the assumption that neural networks can learn to predict the presence or absence of a given concept independently of other concepts. Recent work, however, strongly suggests that this assumption may fail to hold in Concept Bottleneck Models (CBMs), a quintessential family of concept-based interpretable architectures. In this paper, we investigate whether CBMs correctly capture the degree of conditional independence across concepts when such concepts are localised both \textit{spatially}, by having their values entirely defined by a fixed subset of features, and \textit{semantically}, by having their values correlated with only a fixed subset of predefined concepts. To understand locality, we analyse how changes to features outside of a concept's spatial or semantic locality impact concept predictions. Our results suggest that even in well-defined scenarios where the presence of a concept is localised to a fixed feature subspace, or whose semantics are correlated to a small subset of other concepts, CBMs fail to learn this locality. These results cast doubt upon the quality of concept representations learnt by CBMs and strongly suggest that concept-based explanations may be fragile to changes outside their localities.
Do Concept Bottleneck Models Obey Locality?
Naveen Raman Mateo Espinosa Zarlenga Juyeon Heo Mateja Jamnik