DeConvNet, Guided BackProp, LRP, were invented to better understand deep neural networks. We show that these methods do not produce the theoretically correct explanation for a linear model. Yet they are used on multi-layer networks with millions of parameters. This is a cause for concern since linear models are simple neural networks. We argue that explanation methods for neural nets should work reliably in the limit of  simplicity, the linear models. Based on our analysis of linear models we propose a generalization that yields two explanation techniques (PatternNet and PatternAttribution) that are theoretically sound for linear models and produce improved explanations for deep networks.
 Deep learning made a huge impact on a wide variety of applications BID8 BID24 BID9 BID15 BID10 BID18 and recent neural network classifiers have become excellent at detecting relevant signals (e.g., the presence of a cat) contained in input data points such as images by filtering out all other, nonrelevant and distracting components also present in the data. This separation of signal and distractors is achieved by passing the input through many layers with millions of parameters and nonlinear activation functions in between, until finally at the output layer, these models yield a highly condensed version of the signal, e.g. a single number indicating the probability of a cat being in the image.While deep neural networks learn efficient and powerful representations, they are often considered a 'black-box'. In order to better understand classifier decisions and to gain insight into how these models operate, a variety techniques have been proposed BID20 BID25 BID12 BID1 BID0 BID11 BID26 BID22 BID27 BID23 BID21 . These methods for explaining classifier decisions operate under the assumption that it is possible to propagate the condensed output signal back through the classifier to arrive at something that shows how the relevant signal was encoded in the input and thereby explains the classifier decision. Simply put, if the classifier detected a cat, the visualization should point to the cat-relevant aspects of the input image from the perspective of the network. Techniques that are based on this principle include saliency maps from network gradients BID1 BID20 , DeConvNet (Zeiler & Fergus, 2014, DCN) , Guided BackProp (Springenberg et al., 2015, GBP) , Figure 1 : Illustration of explanation approaches. Function and signal approximators visualize the explanation using the original color channels. The attribution is visualized as a heat map of pixelwise contributions to the output Layer-wise Relevance Propagation (Bach et al., 2015, LRP) and the Deep Taylor Decomposition (Montavon et al., 2017, DTD) , Integrated Gradients BID23 and SmoothGrad BID21 .The merit of explanation methods is often demonstrated by applying them to state-of-the-art deep learning models in the context of high dimensional real world data, such as ImageNet, where the provided explanation is intuitive to humans. Unfortunately , theoretical analysis as well as quantitative empirical evaluations of these methods are lacking.Deep neural networks are essentially a composition of linear transformations connected with nonlinear activation functions. Since approaches , such as DeConvNet, Guided BackProp, and LRP, back-propagate the explanations in a layer-wise fashion, it is crucial that the individual linear layers are handled correctly. In this work we show that these gradient-based methods fail to recover the signal even for a single-layer architecture, i.e. a linear model. We argue that therefore they cannot be expected to reliably explain a deep neural network and demonstrate this with quantitative and qualitative experiments. In particular, we provide the following key contributions:• We analyze the performance of existing explanation approaches in the controlled setting of a linear model (Sections 2 and 3).• We categorize explanation methods into three groups -functions, signals and attribution (see Fig. 1 ) -that require fundamentally different interpretations and are complementary in terms of information about the neural network (Section 3).• We propose two novel explanation methods -PatternNet and PatternAttribution -that alleviate shortcomings of current approaches, as discovered during our analysis, and improve explanations in real-world deep neural networks visually and quantitatively (Sections 4 and 5).This presents a step towards a thorough analysis of explanation methods and suggests qualitatively and measurably improved explanations. These are crucial requirements for reliable explanation techniques, in particular in domains, where explanations are not necessarily intuitive, e.g. in health and the sciences BID16 .Notation and scope Scalars are lowercase letters (i), column vectors are bold (u), element-wise multiplication is ( ). The covariance between u and v is cov [u, v] , the covariance of u and i is cov [u, i] . The variance of a scalar random variable i is σ 2 i . Estimates of random variables will have a hat (û) . We analyze neural networks excluding the final soft-max output layer. To allow for analytical treatment, we only consider networks with linear neurons optionally followed by a rectified linear unit (ReLU), max-pooling or soft-max. We analyze linear neurons and nonlinearities independently such that every neuron has its own weight vector. These restrictions are similar to those in the saliency map BID20 , DCN (Zeiler & Fergus, 2014) , GBP (Springenberg Figure 2 : For linear models, i.e., a simple neural network, the weight vector does not explain the signal it detects BID5 . The data x = ya s + a d is color-coded w.r.t. the output y = w T x. Only the signal s = ya s contributes to y. The weight vector w does not agree with the signal direction, since its primary objective is canceling the distractor. Therefore, rotations of the basis vector a d of the distractor with constant signal s lead to rotations of the weight vector (right). et al., 2015), LRP BID0 and DTD BID11 . Without loss of generality, biases are considered constant neurons to enhance clarity. To evaluate the quality of the explanations, we focus on the task of image classification. Nevertheless, our method is not restricted to networks operating on image inputs. We used Theano BID2 and BID4 for our implementation. We restrict the analysis to the well-known ImageNet dataset BID13 using the pre-trained VGG-16 model BID19 . Images were rescaled and cropped to 224x224 pixels. The signal estimators are trained on the first half of the training dataset.The vector v, used to measure the quality of the signal estimator ρ(x) in Eq. FORMULA5 , is optimized on the second half of the training dataset. This enables us to test the signal estimators for generalization. All the results presented here were obtained using the official validation set of 50000 samples. The validation set was not used for training the signal estimators, nor for training the vector v to measure the quality. Consequently our results are obtained on previously unseen data.The linear and the two component signal estimators are obtained by solving their respective closed form solutions (Eq. (4) and Eq. FORMULA15 ). With a highly parallelized implementation using 4 GPUs this could be done in 3-4 hours. This can be considered reasonable given that several days are required to train the actual network. The quality of a signal estimator is assessed with Eq. (1). Solving it with the closed form solution is computationally prohibitive since it must be repeated for every single weight vector in the network. Therefore we optimize the equivalent least-squares problem using stochastic mini-batch gradient descent with ADAM Kingma & Ba (2015) until convergence. This was implemented on a NVIDIA Tesla K40 and took about 24 hours per optimized signal estimator.After learning to explain, individual explanations are computationally cheap since they can be implemented as a back-propagation pass with a modified weight vector. As a result, our method produces explanations at least as fast as the work by BID3 on real time saliency. However, our method has the advantage that it is not only applicable to image models but is a generalization of the theory commonly used in neuroimaging BID5 . Measuring the quality of signal estimators In Fig. 3 we present the results from the correlation measure ρ(x), where higher values are better. We use random directions as baseline signal estimators. Clearly, this approach removes almost no correlation. The filter-based estimator S w succeeds in removing some of the information in the first layer. This indicates that the filters are similar to the patterns in this layer. However, the gradient removes much less information in the higher layers. Overall, it does not perform much better than the random estimator. This implies that the weights do not correspond to the detected stimulus in a neural network. Hence the implicit assumptions about the signal made by DeConvNet and Guided BackProp is not valid. The optimized estimators remove much more of the correlations across the board. For convolutional layers, S a and S a+− perform comparably in all but one layer. The two component estimator S a+− is best in the dense layers.Image degradation The first experiment was a direct measurement of the quality of the signal estimators of individual neurons. The second one is an indirect measurement of the quality, but it considers the whole network. We measure how the prediction (after the soft-max) for the initially selected class changes as a function of corrupting more and more patches based on the ordering assigned by the attribution (see BID14 . This is also related to the work by BID27 . In this experiment, we split the image in non-overlapping patches of 9x9 pixels. We compute the attribution and sum all the values within a patch. We sort the patches in decreasing order based on the aggregate heat map value. In step n = 1..100 we replace the first n patches with the their mean per color channel to remove the information in this patch. Then, we measure how this influences the classifiers output. We use the estimators from the previous experiment to obtain the function-signal attribution heat maps for evaluation. A steeper decay indicates a better heat map.Results are shown in Fig. 4 . The baseline, in which the patches are randomly ordered, performs worst. The linear optimized estimator S a performs quite poorly, followed by the filter-based estimator S w . The trivial signal estimator S x performs just slightly better. However, the two component model S a+− leads to the fastest decrease in confidence in the original prediction by a large margin. Its excellent quantitative performance is also backed up by the visualizations discussed next.Qualitative evaluation In FIG1 , we compare all signal estimators on a single input image. For the trivial estimator S x , the signal is by definition the original input image and, thus, includes the distractor. Therefore, its noisy attribution heat map shows contributions that cancel each other in the neural network. The S w estimator captures some of the structure. The optimized estimator S a results in slightly more structure but struggles on color information and produces dense heat maps. The two component model S a+− on the right captures the original input during signal estimation and produces a crisp heat map of the attribution. FIG2 shows the visualizations for six randomly selected images from ImageNet. PatternNet is able to recover a signal close to the original without having to resort to the inclusion of additional rectifiers in contrast to DeConvNet and Guided BackProp. We argue that this is due to the fact that the optimization of the pattern allows for capturing the important directions in input space. This contrasts with the commonly used methods DeConvNet, Guided BackProp, LRP and DTD, for which the correlation experiment indicates that their implicit signal estimator cannot capture the true signal in the data. Overall, the proposed approach produces the most crisp visualization in addition to being measurably better, as shown in the previous section. Additonally, we also contrast our methods to the prediction-differences analysis by BID27 in the supplementary material.Relation to previous methods Our method can be thought of as a generalization of the work by BID5 , making it applicable on deep neural networks. Remarkably, our proposed approach can solve the toy example in section 2 optimally while none of the previously published methods for deep learning are able to solve this BID0 BID11 BID21 BID23 BID27 BID3 BID26 BID22 . Our method shares the idea that to explain a model properly one has to learn how to explain it with Zintgraf et al. FORMULA5 and BID3 . Furthermore, since our approach is after training just as expensive as a single back-propagation step, it can be applied in a real-time context, which is also possible for the work done by BID3 but not for BID27 . Understanding and explaining nonlinear methods is an important challenge in machine learning. Algorithms for visualizing nonlinear models have emerged but theoretical contributions are scarce. We have shown that the direction of the model gradient does not necessarily provide an estimate for the signal in the data. Instead it reflects the relation between the signal direction and the distracting noise contributions ( Fig. 2) . This implies that popular explanation approaches for neural networks (DeConvNet, Guided BackProp, LRP) do not provide the correct explanation, even for a simple linear model. Our reasoning can be extended to nonlinear models. We have proposed an objective function for neuron-wise explanations. This can be optimized to correct the signal visualizations (PatternNet) and the decomposition methods (PatternAttribution) by taking the data distribution into account. We have demonstrated that our methods constitute a theoretical, qualitative and quantitative improvement towards understanding deep neural networks.
Without learning, it is impossible to explain a machine learning model's decisions.
{"src": "scitldr-AIC_train", "paper_id": "Hkn7CBaTW"}