Hyperparameter Importance. Understanding which hyperparameters influence model performance can provide valuable insights into the tuning strategy [1]}.
To quantify relevance of hyperparameters, models that inherently quantify feature relevance – e.g., GPs with ARD kernel [2]} – can be used as surrogate models.
[3]} quantified the importance of hyperparameters based on a random forest fitted on data generated by BO, for which the importance of both the main and the interaction effects of hyperparameters was calculated by a functional ANOVA approach. Similarly, [4]} quantified the hyperparameter importance of residual neural networks.
These works highlight how useful it is to quantify the importance of hyperparameters.
However, importance scores do not show how a specific hyperparameter affects the model performance according to the surrogate model. Therefore, we propose to visualize the assumed marginal effect of a hyperparameter.
A model-agnostic interpretation method that can be used for this purpose is the PDP.
