The second line of works performs specific perturbations on image regions, such as occlusion, adding noise, inpainting, and blurring.
After observing the effect of a perturbation on the model's output, the methods determine the importance of the region that was perturbed.
Specifically, Occlusion [1]} blocks out square parts of the image in a sliding window manner and captures the drop in the class score to determine the importance.
LIME [2]} approximates the deep model by a linear classifier, trains it in the vicinity of the input point by using samples with occluded superpixels and uses the learned weights as the measure of superpixel importance. The Meaningful Perturbation approach [3]} and Real Time Image Saliency [4]} optimize the perturbation mask using gradient descent. These two works also define themselves as black-box methods, however their definition of “black-box” is different from ours.
While their methods can be applied to any differentiable image classification network, they still require access to model's weights and gradients for gradient descent optimization.
Along with [1]}, [2]}, [7]}, our work uses a stricter definition of “black-box”, entirely prohibiting access to any of the model's internal parameters.
Finally, RISE [7]} generates a set of random masks, applies the classifier to masked versions of the input and uses the predicted class probabilities as weights, computing a weighted sum of the masks as the saliency map. We use this masking technique to explain object detectors rather than image classifiers. Since this and other saliency methods described above cannot be directly applied to the detection task, our work extends the prior state of the art by enabling detector explanation. We describe key differences that have to be addressed for extending saliency methods to object detection in the next section.
