For this reason, a number of different techniques have been proposed to try to make neural networks more explainable. Proposed approaches include extracting interpretable rules [1]}, counterfactual explanations [2]}, [3]}, model distillation [4]}, and feature attribution [5]}, [6]}, [7]}. In this work, we focus on the latter type of explanation. Feature attribution explanations are among the most popular techniques for explaining image classification models, because they can easily be visualized as a heatmap showing which pixels in an image are important (in the case of color images, the attribution value of a pixel can be defined to be the average or maximum absolute value of its three color components). Feature attribution techniques can also be used to measure the importance of hidden neurons or layers [8]}, [6]}, although the focus in this work is on pixel attribution in the image domain.
