@inproceedings{
puli2024explanations,
title={Explanations that reveal all through the definition of encoding},
author={Aahlad Manas Puli and Nhi Nguyen and Rajesh Ranganath},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=mkw6x0OExg}
}
Feature attributions attempt to highlight what inputs drive predictive power. Good attributions or explanations are thus those that produce inputs that retain this predictive power; accordingly, evaluations of explanations score their quality of prediction. However, evaluations produce scores better than what appears possible from the values in the explanation for a class of explanations, called encoding explanations. Probing for encoding remains a challenge because there is no general characterization of what gives the extra predictive power. We develop a deﬁnition of encoding that identiﬁes this extra predictive power via conditional dependence and show that the deﬁnition ﬁts existing examples of encoding. This deﬁnition implies, in contrast to encoding explanations, that non-encoding explanations contain all the informative inputs used to produce the explanation, giving them a “what you see is what you get” property, which makes them transparent and simple to use. Next, we prove that existing scores (ROAR, FRESH, EVAL-X) do not rank non-encoding explanations above encoding ones, and develop STRIPE-X which ranks them correctly. After empirically demonstrating the theoretical insights, we use STRIPE-X to uncover encoding in LLM-generated explanations for predicting the sentiment in movie reviews.
Explanations that reveal all through the deﬁnition of encoding
Aahlad Manas Puli Nhi Nguyen Rajesh Ranganath