Model-agnostic interpretation methods [1]} are a (partial) remedy to the lack of interpretable parameters of more complex models.
Model-agnostic methods follow a general procedure of 1) sampling data, 2) manipulating this data, 3) predicting and 4) aggregating the predictions [2]}.
Since none of these steps depend on specific model properties, model-agnostic interpretation techniques allow us to study the behavior of arbitrary models.
Partial dependence (PD) plots [3]} and permutation feature importance (PFI) [4]}, [5]} are popular model-agnostic methods for describing the relationship between input features and model outcome on a global level.
PD plots visualize the average effect features have on the prediction, and PFI estimates how much each feature improves the model performance and therefore how relevant a feature is.
However, PD and PFI merely describe the prediction (or classification) function, but lack a theory that connects them to the data generating process.
Treating PD and PFI as statistical estimators (like coefficients in a regression model) would require a theoretical counterpart in the DGP: a ground truth estimand that these interpretation methods are supposed to retrieve.
Furthermore, for proper inference about the DGP, we need to quantify the uncertainty of PD and PFI estimators.
Linear regression models, for example, provide variance estimates for the coefficients, which help to distinguish true effects from randomness and allow confidence interval estimation and hypothesis testing.
Most machine learning approaches, however, do not provide variance estimates for their predictions or model parameters.
Yet, the training process itself can be a relevant source of variance as the trained model heavily depends on the specific training data.
