Common explanatory tools, including Local Interpretable Model-Agnostic Explanations (LIME)  [1]} and Shapley Additive Explanations (SHAP)  [2]}, are data-centric.
They assess contributions of individual attribute values to predictive performance of the models. Insights gleaned from such analyses are primarily of confirmatory value; a clinician can confirm that the model pays attention to similar features that she would consider in analyzing her current patient.
However, these tools can be brittle, hide biases,
and do not provide useful diagnostic information about safety of the models.
