
Attribution range: Indicates which parts of the attribution map the metric actually evaluates: we denote metrics that evaluate the most important, least important, or all inputs as high-end, low-end or overall metrics, respectively. For example: \(Del_{MoRF}\)  and \(Del_{LeRF}\)  evaluate the high- and low-end, respectively, because they measure the influence of removing the top and bottom \(k\)  pixels, respectively (see further).

Masking: Indicates whether the metric relies on masking inputs in its implementation. Metrics that do, can be implemented in different ways, as the choice of a neutral value to replace pixels with is not obvious (see Section REF ).

Data type: Indicates which types of data the metric can be applied to. In our case, a metric can either be applied to any kind of data, or only to image data (for example, because it relies on an image segmentation algorithm [1]}, or an adversarial patch [2]}).

Complexity: Indicates the computational complexity of the metric expressed as a number of forward passes through the model. \(C_{mth}\)  is the complexity of the attribution method being evaluated, also expressed as a number of forward/backward passes.

Interface: We define two interfaces for attribution metrics:

Type I: \(M: \mathcal {M} \times \mathcal {X} \times \mathbb {R}^d \times \lbrace 1, \dots , o\rbrace  \rightarrow \mathbb {R}\) . A type I metric accepts an attribution map \(\mathbf {e} \in \mathbb {R}^d\)  to evaluate. This allows one to compute the metric result for any attribution map, regardless of whether the implementation of the attribution method that generated it is available or not.

Type II: \(M: \mathcal {M} \times \mathcal {X} \times \mathcal {E} \times \lbrace 1, \dots , o\rbrace  \rightarrow \mathbb {R}\) . A type II metric needs access to the attribution method under evaluation \(E \in \mathcal {E}\) , because the method needs to be re-executed at some point in the computation of the metric. If the implementation of the attribution method is not available, this type of metric cannot be computed.


<TABLE>