Understanding the flow of information in Deep Neural Networks (DNNs) is a challenging problem that has gain increasing attention over the last few years. While several methods have been proposed to explain network predictions, there have been only a few attempts to compare them from a theoretical perspective. What is more, no exhaustive empirical comparison has been performed in the past. In this work we analyze four gradient-based attribution methods and formally prove conditions of equivalence and approximation between them. By reformulating two of these methods, we construct a unified framework which enables a direct comparison, as well as an easier implementation. Finally, we propose a novel evaluation metric, called Sensitivity-n and test the gradient-based attribution methods alongside with a simple perturbation-based attribution method on several datasets in the domains of image and text classification, using various network architectures. While DNNs have had a large impact on a variety of different tasks BID10 BID8 BID12 BID21 BID28 , explaining their predictions is still challenging. The lack of tools to inspect the behavior of these black-box models makes DNNs less trustable for those domains where interpretability and reliability are crucial, like autonomous driving, medical applications and finance.In this work, we study the problem of assigning an attribution value, sometimes also called "relevance" or "contribution", to each input feature of a network. More formally, consider a DNN that takes an input x = [x 1 , ..., x N ] ∈ R N and produces an output S(x) = [S 1 (x), ..., S C (x)], where C is the total number of output neurons. Given a specific target neuron c, the goal of an attribution method is to determine the contribution R c = [R c 1 , ..., R c N ] ∈ R N of each input feature x i to the output S c . For a classification task, the target neuron of interest is usually the output neuron associated with the correct class for a given sample. When the attributions of all input features are arranged together to have the same shape of the input sample we talk about attribution maps FIG0 , which are usually displayed as heatmaps where red color indicates features that contribute positively to the activation of the target output, and blue color indicates features that have a suppressing effect on it.The problem of finding attributions for deep networks has been tackled in several previous works BID22 BID30 BID24 BID2 BID20 BID25 BID31 . Unfortunately, due to slightly different problem formulations, lack of compatibility with the variety of existing DNN architectures and no common benchmark, a comprehensive comparison is not available. Various new attribution methods have been published in the last few years but we believe a better theoretical understanding of their properties is fundamental. The contribution of this work is twofold:1. We prove that -LRP BID2 and DeepLIFT (Rescale) BID20 can be reformulated as computing backpropagation for a modified gradient function (Section 3). This allows the construction of a unified framework that comprises several gradient-based attribution methods, which reveals how these methods are strongly related, if not equivalent under certain conditions. We also show how this formulation enables a more convenient implementation with modern graph computational libraries.2. We introduce the definition of Sensitivity-n, which generalizes the properties of Completeness BID25 and Summation to Delta BID20 and we compare several methods against this metric on widely adopted datasets and architectures. We show how empirical results support our theoretical findings and propose directions for the usage of the attribution methods analyzed (Section 4). In this work, we have analyzed Gradient * Input, -LRP, Integrated Gradients and DeepLIFT (Rescale) from theoretical and practical perspectives. We have shown that these four methods, despite their apparently different formulation, are strongly related, proving conditions of equivalence or approximation between them. Secondly, by reformulating -LRP and DeepLIFT (Rescale), we have shown how these can be implemented as easy as other gradient-based methods. Finally, we have proposed a metric called Sensitivity-n which helps to uncover properties of existing attribution methods but also traces research directions for more general ones.Nonlinear operations. For a nonlinear operation with a single input of the form x i = f (z i ) (i.e. any nonlinear activation function), the DeepLIFT multiplier (Sec. 3.5.2 in Shrikumar et al. BID20 ) is: DISPLAYFORM0 Nonlinear operations with multiple inputs (eg. 2D pooling) are not addressed in BID20 . For these, we keep the original operations' gradient unmodified as in the DeepLIFT public implementation. By linear model we refer to a model whose target output can be written as S c (x) = i h i (x i ), where all h i are compositions of linear functions. As such, we can write DISPLAYFORM1 for some some a i and b i . If the model is linear only in the restricted domain of a task inputs, the following considerations hold in the domain. We start the proof by showing that, on a linear model, all methods of TAB0 are equivalent.Proof. In the case of Gradient * Input, on a linear model it holds DISPLAYFORM2 , being all other derivatives in the summation zero. Since we are considering a linear model, all nonlinearities f are replaced with the identity function and therefore ∀z : g DL (z) = g LRP (z) = f (z) = 1 and the modified chain-rules for LRP and DeepLIFT reduce to the gradient chain-rule. This proves that -LRP and DeepLIFT with a zero baseline are equivalent to Gradient * Input in the linear case. For Integrated Gradients the gradient term is constant and can be taken out of the integral: DISPLAYFORM3 , which completes the proof the proof of equivalence for the methods in TAB0 in the linear case.If we now consider any subset of n features x S ⊆ x, we have for Occlusion-1: DISPLAYFORM4 where the last equality holds because of the definition of linear model (Equation 9 ). This shows that Occlusion-1, and therefore all other equivalent methods, satisfy Sensitivity-n for all n if the model is linear. If, on the contrary, the model is not linear, there must exists two features x i and x j such that DISPLAYFORM5 . In this case, either Sensitivity-1 or Sensitivity-2 must be violated since all methods assign a single attribution value to x i and x j .
Four existing backpropagation-based attribution methods are fundamentally similar. How to assess it?
{"src": "scitldr-AIC_train", "paper_id": "Sy21R9JAW"}