Recently, [1]} have shown that some of these metrics are very dependent on implementation details, and don't appear to be measuring the same underlying properties of explanations. This is shown by measuring the correlation between different metric scores. The authors find that details such as how pixels are masked (by setting them to 0 vs. replacing them with random noise), or in what order they are masked (by decreasing or increasing importance), have a great influence on the quality scores given by the metric. This suggests that these metrics, although they are all designed to measure the “accuracy” of explanations, appear to be measuring different underlying properties. We build upon this work by applying a similar but more extensive analysis on a larger number of metrics and methods. In doing so, we can draw more global conclusions about how different methods and metrics relate to each other, and which methods and metrics may be most desirable for specific use cases.
