Beyond grid-search and random search [1]} for HPO,
sequential model-based optimization (SMBO) is a common technique with different
`surrogate models' such as Gaussian processes [2]}, random
forests [3]} and tree-parzen estimators [4]}.
Successive halving [5]}, [6]} and HyperBand
[7]} utilize computationally cheap multi-fidelity
approximations of the objective based on some budget (training samples/epochs)
with bandit learning to eliminate unpromising candidates early. These schemes
essentially perform an efficient random search over discrete or
discretized continuous spaces. BOHB [8]} combines SMBO (with TPE) and
HyperBand. Meta-learning [9]}, [10]}, [11]} leverages past experiences with search space refinements
and promising starting points.
SMBO with a large number of variables along with conditional dependencies has
been used to solve CASH in the widely-used Auto-WEKA [12]}, [13]}
and Auto-sklearn [14]} toolkits. Both apply the general
purpose SMAC framework [3]} to find optimal ML pipelines of fixed
shape.
Hyperopt-sklearn [16]} utilizes TPE as the SMBO.
