Recently, a question about the validity of saliency methods has been raised, comparing them to edge detection techniques that do not depend on the model or training data but still produce visually compelling outputs resembling those of saliency methods [1]}.
In this study, it has been shown that the outputs of some widely accepted saliency methods do not change significantly when the weights of the model that each such method claims to explain are randomized. In such cases, a confirmation bias may result in an invalid human assessment when relying exclusively on the visual evaluation. To address these concerns, we perform a model parameter randomization test. Our results confirm that replacing the learned weights with the random ones produces unintelligible saliency maps, meaning that the method relies on the information within the trained model to produce the explanations.
