General metrics for the evaluation of explainers have also been proposed. Some works gauge effectiveness by human evaluation and proxy tasks, including subjective ratings and the improvement of human performance [1]}, [2]}.
Several classes of hand-crafted metrics have been put forward.
Those based on erasure
attempt to evaluate feature importance by removing parts of an input based on the explanation, e.g., under the assumption that higher-weighted features should affect accuracy more [3]}, [4]}.
“Fidelity” metrics
compare surrogate accuracy relative to the model being explained [5]}, [6]}.
Another assumption made is that explanations should be robust, where better explanations are considered to be those that do not fluctuate between like instances [7]}.
Further, some metrics reward monotonic behavior between feature importance and the model response [8]}.
A more extensive survey of evaluation methods is presented in [9]}.
The assumptions that these metrics make prevent them from being entirely objective; accurate explainers are not necessarily faithful, plausible explanations may only be convincing, and unexpected explanations may be due to the model rather than the explainer.
On the contrary, our work makes no such assumptions about the model nor explainer.
