Deep neural networks have achieved great success in various fields, such as computer vision and natural language processing.
However, understanding and interpreting the decision processes of black-box models, such as neural networks, are notoriously difficult [1]}.
Therefore, their application is limited to high-stakes domains, such as healthcare and social safety.
Therefore, explainable artificial intelligence (AI) methods to explain the behaviors of black box models have been widely studied in recent years.
Local interpretable model-agnostic explanations (LIME) [2]} is a representative explainable AI method.
It attempts to locally approximate a black-box model using an interpretable model to explain individual predictions.
LIME can explain a local area of the black-box model if the interpretable model has approximated the black-box model to be explained well.
However, LIME often fails to explain the global behavior of the black box model.
Shapley additive explanation (SHAP) [3]} is a game-theoretic approach for model-agnostic explanations.
SHAP generates predictions of a black-box model based on possible combinations of input features and obtains the effect of a feature by exploiting changes in predictions when the feature is dropped.
However, the explanation of SHAP may not be faithful to the actual predictions of the black-box model.
As mentioned above, explainable AI methods are still limited in their applications to high-stakes domains because they often fail to explain and may not be faithful to the actual behaviors of black-box models; thus, we need interpretable AI methods [4]}.
