
Metric scores vary strongly for different datasets. This implies that the performance of attribution methods should be measured for each specific use case, rather than drawing general conclusions from the results on a set of benchmark datasets.

Most metrics tend to have low ranking consistency, shown by the relatively low values of Krippendorff \(\alpha \) . From this we conclude that a statistical testing approach should be used to draw any dataset-wide conclusions. Also, the ranking consistency values of metrics themselves are not consistent across datasets, implying that there is no generally superior metric.

We extend the conclusion from [1]} that metrics don't necessarily measure the same underlying concept to a larger amount of metrics, including Sensitivity-n [2]}, Infidelity [3]}, IROF [4]} and Impact Coverage [5]}. This can be seen in the low inter-metric correlation values between these metrics.

Finally, we also introduce Seg-Sensitivity-n as an extension of Sensitivity-n [2]}, and show that it has a higher signal-to-noise ratio than Sensitivity-n on high-dimensional datasets.

