Furthermore, the interpretability of machine learning models, as explaining which and how input variables contribute to the predictive outcomes, is crucial to the clinicians. A good explanation helps clinicians understand the risk factors, thus knowing how to plan for and deliver postoperative care and timely interventions. Despite the invention of model-agnostic explanation methods [1]}, [2]}, attribution methods tailored for deep models [3]}, [4]}, and self-explaining models [5]}, [6]}, [7]}, [8]}, [9]}, [10]}, it remains challenging to generate accurate explanations identifying important data segments in fine-grained time series that can benefit clinicians and medical research.
