Multi-Objective Hyperparameter Optimization -- An Overview
Hyperparameter optimization constitutes a large part of typical modern
machine learning workflows. This arises from the fact that machine learning
methods and corresponding preprocessing steps often only yield optimal
performance when hyperparameters are properly tuned. But in many applications,
we are not only interested in optimizing ML pipelines solely for predictive
accuracy; additional metrics or constraints must be considered when determining
an optimal configuration, resulting in a multi-objective optimization problem.
This is often neglected in practice, due to a lack of knowledge and readily
available software implementations for multi-objective hyperparameter
optimization. In this work, we introduce the reader to the basics of multi-
objective hyperparameter optimization and motivate its usefulness in applied
ML. Furthermore, we provide an extensive survey of existing optimization
strategies, both from the domain of evolutionary algorithms and Bayesian
optimization. We illustrate the utility of MOO in several specific ML
applications, considering objectives such as operating conditions, prediction
time, sparseness, fairness, interpretability and robustness.