Attribution maps have been typically used for singular classification outputs, and this is be one of the first applications of this technique to a model with a large multi-dimensional output. McGovern et al. [1]} discuss various techniques for explaining black box ML models on various weather prediction tasks. However, all of the applications involve a single predicted output for each class. In our case, while we can obtain the attributions for each predicted pixel that represents a \(1~\textrm {km} \times 1~\textrm {km}\) , it will be humanly impossible to consume and make sense of the attribution maps for each pixel in the output separately. Additionally, the complexity of most of the existing interpretability tools increases greatly if they were to be applied to every single predicted pixel in a multi-dimensional output, separately. We, therefore, use Integrated Gradients to explain not just a single predicted pixel but the entire multi-dimensional output of predictions for the given input. For global interpretability, we aggregate per-feature attributions across space and over all the predictions and samples. This essentially gives us a single attribution value for each feature that represents the relative importance of that feature to the model.
