
Baseline selection: First, a baseline attribution method must be defined. In general, a uniform random baseline can be used, but more specific baselines can also be chosen depending on the use case (for example, an edge detector).

Metric selection: Next, a selection of metric implementations must be made. This can be done manually, if such a selection of metrics is obvious from the use case and there is a clear approach to masking available (for example, if all images have a constant background color), or using a pilot study. In such a pilot study, a large number of metrics and masking approaches are tested on a limited number of images. We then recommend computing inter-metric correlations and Krippendorff \(\alpha \)  values, and selecting those metric implementations that have high Krippendorff \(\alpha \)  and low inter-metric correlations. In this way, a minimal number of images can used to draw dataset-wide conclusions, and metric scores will contain a minimal amount of redundant information.

Rough statistical analysis: Once the metric scores are computed for all methods and the baseline, a rough overview of method performance can be made using a pairwise, non-parametric statistical test (such as a Wilcoxon signed rank test). We specifically recommend this test because metric scores are dependent on the specific image being used [1]}, and metric scores don't necessarily follow a normal distribution.

Detailed comparison: Using the rough overview made in the previous step, we recommend selecting a smaller number of well-performing attribution methods, if possible with varying computational complexity. Those methods can then be compared in more detail using new pairwise Wilcoxon tests, this time between the two methods rather than between a method and the baseline. The CLES can then be used to assess the fraction of cases where one method is superior to another. Based on these results, as well as the complexity of the methods being considered, a final selection of an ideal method can then be made.

<FIGURE>