Hyper-parameter optimization (HPO) for a single machine learning (ML)
algorithm is widely studied in AutoML [1]}, [2]}. HPO was generalized to the Combined Algorithm Selection and HPO (CASH) problem to configure multiple
stages of a ML pipeline (transformers, feature selectors, predictive models)
automatically [3]}, [4]}. Since the CASH formulation,
various innovative solvers have been
proposed [5]}, [6]}, [7]}, [8]}.
CASH has two main challenges: (i) the tight coupling between the algorithm
selection & HPO; and (ii) the black-box nature of optimization objective
lacking any explicit gradients – feedback is only available in the form of
(often expensive) function evaluations. Furthermore, the CASH formulation does
not explicitly handle constraints on the individual ML pipelines
such as fairness or domain-specific constraints on individual ML pipelines.
