Geometrically Guided Integrated Gradients

Interpretability methods for deep neural networks mainly focus on the sensitivity of the class score with respect to the original or perturbed input, usually measured using actual or modified gradients. Some methods also use a model-agnostic approach to understanding the rationale behind every prediction. In this paper, we argue and demonstrate that local geometry of the model parameter space relative to the input can also be beneficial for improved post-hoc explanations. To achieve this goal, we introduce an interpretability method called"geometrically-guided integrated gradients"that builds on top of the gradient calculation along a linear path as traditionally used in integrated gradient methods. However, instead of integrating gradient information, our method explores the model's dynamic behavior from multiple scaled versions of the input and captures the best possible attribution for each input. We demonstrate through extensive experiments that the proposed approach outperforms vanilla and integrated gradients in subjective and quantitative assessment. We also propose a"model perturbation"sanity check to complement the traditionally used"model randomization"test.


Introduction
The past decade has seen a wealth of new advancements in deep learning (DL), improving performance in a wide array of possible problems in many areas, especially in classification tasks from computer vision and natural language processing [Chai et al., 2021, Young et al., 2018. However, this improved performance comes with a cost, the models are not easily explained or described with simple observations. This lack of intelligibility hinders wide-scale use of these models in safetycritical domains such as healthcare, education, the justice system, and many others. This need for decipherability has given way to parallel advancements in a subfield of machine learning known as interpretability. Interpretability has grown as quickly as the DL field itself, providing many new state-of-the-art techniques [Baehrens et al., 2010, Simonyan et al., 2013, Shrikumar et al., 2017, Sundararajan et al., 2017, Selvaraju et al., 2017, Bach et al., 2015, Montavon et al., 2017, Kapishnikov et al., 2019 Interpretability research generally tackles these problems by either describing the model's decisionmaking processes or by generating post-hoc explanations, either for the model as a whole, or for each sample. There are many post-hoc explanation methods from research spanning the last 2 decades. The performance and properties of these methods vary widely across different architectures and domains. In recent years, beginning with integrated gradients (IG) [Sundararajan et al., 2017], some of these methods [Kapishnikov et al., 2019, Xu et al., 2020 have satisfied two important properties: sensitivity and implementation invariance.
These post-hoc analysis methods, specifically the ones that leverage model gradients to explain each sample, are known as saliency techniques. Vanilla gradients (GRAD) [Baehrens et al., 2010, Simonyan et al., 2013 and IG, two of the most popular methods, can be noisy [Montavon et al., 2017, Samek et al., 2016, Smilkov et al., 2017, Sturmfels et al., 2020. We show an example of this noise in Figure 1. GRAD specifically violates both implementation invariance and sensitivity. IG also has specific noise relating to the averaging or integrating over the interpolation path. Other sources include the large curvature of the network's decision function [Dombrowski et al., 2019], numerical approximation of integration [Kapishnikov et al., 2021], and baseline choices [Xu et al., 2020, Sturmfels et al., 2020. In order to both conform to the two aforementioned properties and reduce noise, we suggest that the gradients can be improved by leveraging more aspects of the model space than the exact sample loss or a linear interpolation of the sample as is found in IG. Specifically, geometric properties of the loss landscape, which have previously been utilized to strengthen model performance and robustness , Gotmare et al., 2018, Entezari et al., 2021.
In this paper, we propose Geometrically Guided Integrated Gradients (GGIG), an algorithm that builds on top of traditional IG to reduce noise. We expand IG by ascending through the loss space by maximizing the class-specific logit, much the way class activation maximization [Couteaux et al., 2019] works. This ascension, or maximization of the class logit, allows GGIG to find the gradients that are most discriminative, which we suggest is a valuable property for any interpretability mechanism. After describing the method, we empirically show that GGIG improves the quality and robustness of the saliency maps. Figure 2 describes the general pipeline of the work.

Input
Integrated Gradients Gradients GGIG Figure 1: Comparison of gradients, integrated gradients and GGIG based on top 5% pixel attributions for the "gazelle" prediction. In GGIG explanation, the details of the animal are clearly noticeable and discarded the undesirable regions from the explanation.
Our main contributions are as follows: • We propose an interpretability method, called Geometrically Guided Integrated Gradients, that starts with linear path as used in IG and finds the path that enhances the class activation for the underlying prediction.
• We also propose a model perturbation sanity check, also called σ-perturbation that we think all explanation methods should satisfy.
• We show that the proposed method offers better saliency maps for different datasets and architectures when assessed through visual inspection and quantitative metrics.
GradCAM [Selvaraju et al., 2017] identifies the focal regions pretty well, and maps are highly predictive. However, the saliency maps are blobby [Kapishnikov et al., 2019] and usually expand around the actual objects. While IG has its own specific problems, several recent studies have refined IG attributions because this method has many desirable properties. Kapishnikov et al. [2019] proposed a region-based attribution method, called XRAI, that mainly refines the IG attribution based on attribution density. However, XRAI requires a way to cluster the input features, which may not be available for different data modalities. Kapishnikov et al. [2021] proposed another method, called GIG, which provides an adaptive path method based on input, baseline, and the model. This method starts at the baseline and selects only those pixels with the lowest partial derivatives to take closer to the next interpolation point, thus avoiding the gradient accumulation from saturation regions. In other words, it constitutes the path based on dynamic projections of the linear interpolation path.
To reduce the inherent noise in the saliency maps, we may utilize some useful loss landscape properties as observed in several studies , Gotmare et al., 2018, Entezari et al., 2021 to design reliable interpretability methods.

Explanation Generation Explanation Evaluation
Class Prediction Model GGIG Estimator Figure 2: Overall pipeline of the work. First, we generate explanations for each prediction. Secondly, for quantitative evaluation, we start with a complete defocused image and combine with only salient pixels (thresholded at x%) to create a test image. Finally, we feed the model with the test image and measure both accuracy and relative softmax scores.

Geometrically Guided Integrated Gradients (GGIG)
We propose a method called GGIG, which incorporates the idea of path methods [Sundararajan et al., 2017] and enhances the quality of the attribution by analyzing the local loss behavior. Figure 3 shows the schematic diagram illustrating the functional mechanism of GGIG.
Like IG, GGIG starts from the baseline x and constitutes a linearly interpolated path. However, instead of accumulating gradients along the path, it updates the path gradients in the direction where the model maximizes class activations. GGIG thus maximizes class activation for each of the linearly interpolated points. The procedural steps for GGIG are shown in Algorithm 1. We hypothesize that the prediction curve in the vicinity of x holds important information about the interaction between model f and input x.
Let F : R d → R be defined as the mapping from the input space to the class-specific logit and b 0 , . . . , b n be a set of n linearly interpolated points for each sample between the baseline, b 0 and the exact sample for k ← 0 to n do 4: return e e is GGIG explanation 12: end procedure where j is the incremental ascent over the sample and lr is the learning rate. This ascending mechanism allows the model to take the direction where logit values are enhanced. All gradients over the ascension, ∇F (b j i ), are collected, and the max ∇F (b j i )[p] (for each pixel p), where i = 0, 1, 2, . . . , n and j = 0, 1, 2, . . . , m, is selected as the final attribution for the pixel p.
Logit for the prediction : input : baseline Figure 3: The basic idea of "Geometrically Guided Integrated Gradients." It takes a baseline x , creates a linearly spaced path to the actual input x. From each interpolation point, it follows a trajectory in the loss landscape that maximizes class activation.
IG [Sundararajan et al., 2017] has many desirbale properties and we have used IG as the baseline to compare the proposed method. We used slightly different formulation of IG. Generally, IG uses interpolation technique to integrate importance at different discrete intervals between uninformative baseline, sayx and the input x, to give an integrated estimate of feature importance. IG based feature importance is computed as: The ultimate estimate e depends on the value of k (number of intervals) and the choice of a suitable uninformative baselinex. The traditional integrated gradients scale raw attributions (operand on the right of the multiplication operator) by x − x (operand on the left).
Element-wise multiplication is misleading: Adebayo et al. [2018b] observed that element-wise multiplication could be misleading. This misleading happens mainly because the input dominates the product even with drastic changes in gradient vectors. So, the interpretability methods with this element-wise multiplication component in their formulation can provide input-dominant explanations that may deceive human understanding.
Furthermore, Ancona et al. [2019] suggested that this point-wise multiplication was initially justified to sharpen the gradient explanations; however, it is better justified when a measure of salience is a priority over mere sensitivity. In this paper, we were more interested in the sensitivity of features rather than their marginal salience to the target score. For all of these valid reasons, we did not multiply the integrated gradient with x − x to avoid input dominance from the explanations. Moreover, we did not consider other gradient-based methods like grad input for the same reason.
The baseline can be problematic for correct attributions [Kapishnikov et al., 2019, Sturmfels et al., 2020. Kapishnikov et al. [2019] addressed the issue of baseline by using both black (0, 0, 0) and white (1, 1, 1) baselines. While baseline may be an issue for traditional IG formulation, avoiding direct or modified (as in IG) element-wise multiplication resolves the baseline issue from this work.

Model Perturbation Sanity Check
In this section, we propose a model perturbation sanity check for attribution methods. We refer to this perturbation as σ-perturbation. This perturbation seeks two important properties: We claim that all attribution methods should satisfy this sanity check to ensure their meaningfulness and sensitivity to the model parameters. We applied this sanity check to vanilla gradients, integrated gradients and GGIG. Interestingly, all of the methods satisfied the σ-perturbation sanity check.
We report σ-perturbation sanity check results in Figure 4.
Explanations generated with model perturbations Relative Softmax Score Figure 4: Left: Comparison of saliency-focused images thresholded at 10% while we perturb the model. Right: Relative softmax score during model perturbation. As expected, the similarity with the original explanations breaks and softmax score drops with gradual increase in perturbation level.

Experiments
Training on MNIST Dataset For MNIST [LeCun, 1998], the model architecture was a CNN that consisted of two convolutional layers with (32, 64) filters of sizes (5, 5). Each convolutional layer is followed by a 2 × 2 max pooling layer and a ReLU activation. We fed the final convolution output to a fully connected network with 1024 input and 10 output units (softmax). We optimized the model using stochastic gradient descent (SGD) with a learning rate of 0.0004 and momentum of 0.9. The model was trained for 400 iterations with a mini-batch size of 64 and finally achieved an accuracy of 99.2%.
Post hoc explanation experiments For GGIG, we used a learning rate of 0.0001 for gradient ascent from each linear interpolation point between input x and baseline x = 0. For MNIST, we iterated the gradient ascent for 200 steps and noted the maximum sensitivity along the gradient ascent trajectory for each input. We display the saliency maps obtained on the MNIST dataset in Figure 11.
Quantitative Evaluation on MNIST Dataset Visual inspection of explanation methods can be unreliable as it is possible to create adversarial samples [Goodfellow et al., 2014, Szegedy et al., 2013] that can fool the human eye, totally changing the model predictions. We perform different similarity measures between the maps and the input to understand the quality of the proposed methods on MNIST, namely, Spearman Rank Correlation, Weighted Jaccard Similarity, Structural Similarity, and Normalized (Reverse) Mean Square Error. For quantitative evaluation, both data and maps were rescaled in the range [0, 1]. We assumed that the amount of information a method can capture about the structure and distribution of the input in the saliency maps directly determines its quality as an explanation method. We report the quantitative evaluation in Table 1. Figure 11 in Appendix shows sample maps and the detailed results of quantitative evaluation. As we can observe, GGIG outperforms other methods. Comparatively, GRAD and IG retain little information about the numerical and structural association to the input. Experiments on ImageNet: We also evaluated the proposed methods using a subset of images from the ImageNet dataset [Krizhevsky et al., 2012] and different pretrained models, namely ResNet-101 [He et al., 2016], and Inception V3 [Szegedy et al., 2016]. Though we found meaningful maps in every case, maps still vary in quality possibly due to their architectural differences. Figure 5 and Figure 6 show some maps (more results are in Appendix) produced using different saliency methods and pretrained models. Figure 5: Sample saliency maps for ImageNet Validation set and ResNet101 model generated by gradients, integrated gradients, geometrically guided integrated gradients, and Edge Detector.

Model Randomization Test:
We also performed a Model Randomization test [Adebayo et al., 2018b] (Figure 7) to verify the sensitivity of the methods to the model parameters. To this end, we randomly reinitialized the weights and generated post-hoc explanations using the randomized model. The proposed method GGIG is as sensitive as GRAD and IG, suggesting that our method is highly sensitive to model parameters.
Saliency methods are not edge detectors: Adebayo et al. [2018b] observed that many saliency methods, including vanilla gradients and integrated gradients can appear like edge-detectors for 1-Layer Sum-Pool Conv Model. This assumption may only hold for shallow models. To analyze the edge-detector like behavior for deep models, we conducted an experiment where we replaced the original background of the images with a sharp-changing image. In particular, we assigned ImageNet samples a very different fixed background (a black and white checkerboard) and generated post-hoc explanations using GRAD, IG, and GGIG. We show the resulting explanations in Figure 8. It is obvious from the resulting maps that the model used concepts, not merely edges from the training objects. Moreover, as expected, all the saliency methods ignored the background and only attributed the object's parts for prediction explanations. It is apparent that GGIG method retained learned concepts more accurately during post-hoc explanations. We think that for well-trained deep models, saliency methods no-longer function like an edge-detectors. Figure 6: Comparative saliency maps for Inception V3 model and ImageNet dataset generated by GRAD, IG, GGIG, and Edge Detector. The maps obtained using GGIG are more discriminative and more clearly reveal the underlying structure of the class-associated objects.

Evaluation of Attribution Quality
Visual Analysis: We show few sample explanations thresholded at 10% in Figure 9. As we can observe, GGIG provides the least noisy explanations compared to other methods. As expected, the edge detector does not pay attention to discriminative regions. Rather, it captures only the sharp changes throughout the image. While GRAD and IG attribute lots of redundant or unexpected parts of the image, GGIG directly points to the most discriminative pixels of the image. Moreover, GGIG mostly avoids sharp changes in the image (see the explanation for the "leopard"), while GRAD and IG are highly susceptible to the edges. Figure 9: Comparison of saliency-focused images thresholded at 10%. GGIG, compared to other methods, is more convincing from the human standpoint because it directly points to the discriminative parts of the image while ignoring redundant and unexpected regions.
Quantitative Evaluation: It is challenging to evaluate an interpretability method because of the lack of ground truth saliency or consensus metrics for proper evaluation. Several studies proposed different measures, such as Remove And Retrain (ROAR) [Hooker et al., 2019], Retain And Retrain (RAR) [Rahman et al., 2022], Accuracy Information Curves, Softmax Information Curves [Kapishnikov et al., 2019] to assess the quality of explanations.
The ROAR approach modifies the dataset by removing the features that received top attribution values from each sample. In practice, if the training data have sufficient (redundant) discriminative features [Sturmfels et al., 2020], even after removing a significant number of features, the performance of the retrained model does not drop noticeably. In that scenario, ROAR fails to evaluate the feature relevance correctly. Retain and Retrain (RAR) method resolves the problem by retaining only the critical features instead of removing them. However, both ROAR and RAR methods are time-consuming as they require full retraining of the model. Dabkowski and Gal [2017] proposed a metric, called Smallest Sufficient Regions (SSR), based on the notion of the smallest sufficient region capable of correct prediction. However, this metric is not suitable if the model is susceptible to the scale and aspect ratio of the object. Moreover, as this metric depends on rectangular cropping and reports results as a function of the cropped area, this approach highly penalizes if the saliency map is coherently sparse [Kapishnikov et al., 2019]. Because, in that case, it may span a larger area of the image than the map, which is locally dense, even with the same number of pixels. SSR also imposes a severe challenge because masking creates a sharp boundary between the masked and salient region, causing an out-of-distribution problem for the model. Kapishnikov et al. [2019] proposed two metrics, called Accuracy Information Curve and Softmaxe Information Curve, collectively called Performance Information Curve (PIC), to evaluate the saliency maps with the minimal out-of-distribution setting. We used SIC and AIC to evaluate our method and adopted a similar setup as used in [Kapishnikov et al., 2019]. However, instead of using compressed image size as a proxy information level, we directly computed the entropy of the gray version of the saliency-focused image (test image).
We show the quantitative evaluation results in Figure 10. We started with a complete defocused image and gradually added salient pixels to form a saliency-focused image. We measure the entropy of the saliency-focused image relative to the complete blurred image and call it a normalized entropy, which forms the x-axis. We feed saliency-focused images to the original model and report median softmax scores and accuracy calculated over all images, which forms the y-axis. We show the evaluation procedure in Figure 2.  [Szegedy et al., 2016] and ResNet101 [He et al., 2016] on ImageNet test and validation images respectively Right: Evaluation using Accuracy Information Curve (AIC) metric for the same models and datasets.

Conclusion
In this paper, we propose an interpretability method called GGIG that leverages the model's dynamic behavior in parameter space to enhance the quality of the feature attributions. GGIG starts with a linearly interpolated path as IG and uses gradient ascent to explore how the model interacts with input in the neighborhood. We believe that by looking into the parameter space from different interpolation points, GGIG captures important information about model vs. input interaction. We show the effectiveness of the proposed method through visual inspection and several quantitative metrics across some popular architectures and datasets. As demonstrated, we expect that this work provides insights toward building a more robust interpretability method through model and input interaction.
A Quantitative Evaluation of GGIG on MNIST Dataset Figure 11: Top: Selected maps for MNIST samples generated using different methods. Bottom: Quantitative evaluations using different correlation and similarity metrics. It is obvious that the maps produced by GGIG have higher structural and numerical correlations with the input. Precisely, GGIG, as an explanation method, performs the best by a large quantitative margin.

B Saliency Maps (ImageNet Dataset)
B.1 Architecture: ResNet-101 Figure 12: Explanations (saliency maps) of ResNet-101 predictions for ImageNet samples generated by GRAD, IG, and GGIG methods. As we can see, the GGIG maps have strong correlation and similarity with the corresponding input images. Figure 13: Visualization of saliency maps obtained from Resnet-101 using 'gray' color scheme.
B.2 Architecture: Inception V3 Figure 14: Visualization of saliency maps using 'inferno' color scheme for selected ImageNet samples. It is evident that the GGIG method consistently captures the useful concepts as used by the model for predictions. Figure 15: Visualization of saliency maps using 'gray' color scheme for selected ImageNet samples. Figure 16: Visualization of saliency maps for selected ImageNet samples using 'coolwarm' color scheme. Figure 17: Visualization of saliency maps for selected ImageNet samples using 'Reds' color scheme. Figure 18: Visualization of saliency maps for selected ImageNet samples using 'reverse bwr' color scheme.