We propose to analyze surrogate models with methods from IML to provide insights into the results of HPO.
In the context of BO, typical choices for surrogate models are flexible, probabilistic black-box models, such as Gaussian processes (GP) or random forests.
Interpreting the effect of single hyperparameters on the performance of the model to be tuned is analogous to interpreting the feature effect of the black-box surrogate model.
We focus on the PDP [1]}, which is a widely-used methodThere exist various implementations [2]}, [3]}), extensions [4]}, [5]} and applications [6]}, [7]}. to visualize the average marginal effect of single features on a black-box model's prediction.
When applied to surrogate models, they provide information on how a specific hyperparameter influences the estimated model performance. However, applying PDPs out of the box to surrogate models might lead to misleading conclusions.
Efficient optimizers such as BO tend to focus on exploiting promising regions of the hyperparameter space while leaving other regions less explored. Therefore, a sampling bias in input space is introduced, which in turn can lead to a poor fit and biased interpretations in underexplored regions of the space.
