Saliency methods aim to explain the predictions of deep neural networks. These methods lack reliability when the explanation is sensitive to factors that do not contribute to the model prediction. We use a simple and common pre-processing step ---adding a mean shift to the input data--- to show that a transformation with no effect on the model can cause numerous methods to incorrectly attribute. We define input invariance as the requirement that a saliency method mirror the sensitivity of the model with respect to transformations of the input. We show, through several examples, that saliency methods that do not satisfy a input invariance property are unreliable and can lead to misleading and inaccurate attribution. While considerable research has focused on discerning the decision process of neural networks BID1 BID9 BID2 BID15 BID12 BID0 BID14 BID6 BID5 BID16 BID13 BID11 BID4 , there remains a trade-off between model complexity and interpretability. Research to address this tension is urgently needed; reliable explanations build trust with users, helps identify points of model failure, and removes barriers to entry for the deployment of deep neural networks in domains high stakes like health care and others.In deep models, data representation is delegated to the model. We cannot generally say in an informative way what led to a model prediction. Instead, saliency methods aim to infer insights about the f (x) learnt by the model by ranking the explanatory power of constituent inputs. While unified in purpose, these methods are surprisingly divergent and non-overlapping in outcome. Evaluating the reliability of these methods is complicated by a lack of ground truth, as ground truth would depend upon full transparency into how a model arrives at a decision -the very problem we are trying to solve for in the first place BID13 BID4 .Given the need for a quantitative method of comparison, several properties such as completeness BID0 BID13 , implementation invariance and sensitivity BID13 have been articulated as desirable to ensure that saliency methods are reliable. Implementation invariance, proposed as an axiom for attribution methods by BID13 , is the requirement that functionally equivalent networks (models with different architectures but equal outputs all inputs), always attribute in an identical way.This work posits that a second invariance axiom, which we term input invariance, needs to be satisfied to ensure reliable interpretation of input contribution to the model prediction. Input invariance requires that the saliency method mirror the sensitivity of the model with respect to transformations of the input. We demonstrate that numerous methods do not satisfy input invariance using a simple transformation -mean shifts of the input-that does not affect model prediction or weights. We limit our treatment of input invariance to showing that there exist cases where this property is not satisfied and welcome future research on a broader treatment of this topic.In this work we:• introduce the axiom input invariance and demonstrate that certain saliency methods do not satisfy this property when considering a simple mean shift in the input. (See FIG3 ).• show that when input invariance is missing, the saliency method becomes unreliable and misleading. Using two example reference points for each method we demonstrate that changing the reference causes the attribution to diverge. The attributions are visualized by multiplying them with the input image as is done in the IG paper 1 BID13 . Visualisations were made on ImageNet BID7 and the VGG16 architecture BID9 .• demonstrate that "reference point" methods-Integrated gradients and the Deep Taylor Decomposition-have diverging attribution and input invariance breaking points that depends upon the choice of reference FIG0 .In Section 2, we detail our experiment framework. In Section 3, we determine that while the model is invariant to the input transformation considered, several saliency methods attribute to the mean shift. In Section 4 we discuss "reference point " methods and illustrate the importance of choosing an appropriate reference before discussing some directions of future research in Section 5. Saliency methods are powerful tools to gain intuition about our model. We consider some examples that can cause a break in the reliability of these methods. We show that we are able to purposefully create a deceptive explanation of the network using a hand drawn kitten image.We introduce input invariance as a prerequisite for reliable attribution. Our treatment of input invariance is restricted to demonstrating there is at least one input transformation that causes attribution to fail. We hope this work drives further discussion on this subject. We also acknowledge that saliency methods may still provide intuition for image recognition tasks even if they are not input invariant. Our work is motivated in part because while we can visually inspect for catasthropic attribution failure in images, other modalities (like audio or word vectors) are more opaque and prone to unintentional misrepresentation.
Attribution can sometimes be misleading
{"src": "scitldr-AIC_train", "paper_id": "r1Oen--RW"}