The ROAR (RemOve and Retrain) framework introduced by [1]} offers a helpful way to evaluate such importance measures. This work provides a retraining-based benchmark for evaluating NN attribution/saliency methods by removing variables in order of estimated importance and measuring the drop in predictive power. This work finds that many common attribution methods are no more informative than a random baseline, and aruges that retraining the network after dropping out variables is key in understanding this behavior.
