Several approaches have been proposed for interpreting the predictions made by machine learning models, including model-agnostic approaches and feature attribution approaches designed for deep models. Model-agnostic explanation approaches, such as LIME [1]} and SHAP [2]}, provide general frameworks for different models while treating them as black-box models. There are also feature attribution approaches designed for interpreting neural networks [3]}, [4]}, [5]}, [6]}. Deep models are not always black boxes. When properly designed attention models can be explainable by itself. Self-explaining models allow predictions be interpreted using attention matrices directly [7]}, [8]}, [9]}, [10]}, [11]}. In particular, RAIM [7]}, HiTANet [9]} and STAM [11]} are self-explaining attention models designed for interpreting clinical outcome predictions. Alvarez-Melis et al. propose self-explaining neural networks (SENN) [15]} that have relevance parametrizers for interpretability, which can be optimized jointly with the classification objective. However, these self-explaining deep models are not interpretable end-to-end. In the aforementioned models, the explanations are generated for concept bases [15]} or intermediate representations [7]}, [9]}, [11]}, instead of raw inputs. The concepts bases [15]} or intermediate representations [7]}, [9]}, [11]} do not necessarily reflect the contributions of raw inputs to the predictive outcomes due to the non-linear transformation from the raw inputs to concepts bases [15]} or intermediate representations [7]}, [9]}, [11]}.
