A large number of attribution methods have been developed relying on the gradient of the decision studied.
The first method was introduced in  [1]} and improved in  [2]}, [3]}, [4]} and consists of explaining the decisions of a convolution model by back-propagating the gradient from the output to the input. The resulting gradient heatmap, also called the saliency map, indicates which pixels affect the decision score the most.
However, this family of methods is limited because they focus on the influence of individual pixels in an infinitesimal neighborhood around the input image in the image space. For instance, it has been shown that gradients often vanish when the prediction score to be explained is near the maximum value [5]}.
Integrated Gradient [5]} and SmoothGrad [7]} partially address this issue by accumulating gradients. Another family of attribution methods relies on the neural network's activations. Popular examples include CAM [8]}, which computes an attribution score based on a weighted sum of feature channel activities – right before the classification layer.
GradCAM [9]} extends CAM via the use of gradients, reweighting each feature channel to take into account their importance for the predicted class. Nevertheless, the choice of the layer has a huge impact on the quality of the explanation. In comparison our proposed approach is model-agnostic and hence does not require access to internal computations.
