The shortcomings of these explainers are due to their underlying assumptions which do not hold true for the majority of models, especially the assumptions of independent features and local linearity.
These assumptions are further impacted by the explainer hyperparameters, such as the kernel width for LIME or the background summarization method for SHAP. Tuning of these parameters is dependent upon the data and model. In practice, these knobs can be adjusted until the explanations “look right,” which is not realistic when the most faithful hyperparameters need to be derived from the black box itself.
This is especially troubling as studies show that data scientists overtrust interpretability techniques [1]}. With the results of our study, even those practitioners who do not abuse these explanation tools may still be mislead.
