Over the last decade, advances in automated detection of diseases using deep learning methods have resulted in performance comparable to human observers in multiple domains e.g., retinal image diagnosis [1]}, [2]}, [3]}. The adoption of these methods in clinics is slow if not negligible The key factors hindering trust from end-users, regulators, and patients on deep learning methods in medical imaging are the opaqueness of the algorithms and the tendency to make wrong decisions on complex cases [4]}, [5]}. There is a need to describe the factors used for making a decision and separating samples with higher uncertainty for expert inspection. The former is the domain of explaining the models and is discussed extensively in the literature [6]}, [7]}, [8]}. Studies have been published using a set of explainability methods (called attributions) for retinal image diagnosis [9]}, [10]}, [11]}, [12]}, [13]}. The attribution methods represent the model output as a sum of the contributions from each input feature. This can be visualized as heatmaps for images and various plots for other data [14]}.
