A possible problem with the metrics mentioned above is the fact that masking inputs in images can introduce high-frequency artifacts, which can push the images outside of their normal data distribution. This can cause the model to produce arbitrary outputs. Although the exact impact of this problem on the scores produced by metrics is unclear, some efforts to resolving it have already been made, including the Remove And Retrain (ROAR) procedure [1]}. Here, the authors attempt to resolve the OOD problem by modifying the Deletion metric by [2]} such that after every iteration, the model is retrained on the data where the top \(k\)  pixels are removed. The reasoning is that in this way, the model learns to regard the mean-valued pixels as uninformative.
