We assess the sampling bias of the optimization design points by comparing their empirical distribution to a uniform distribution via Maximum Mean Discrepancy (MMD) [1]}, [2]}, which is covered in more detail in the Appendix REF .
We measure the reliability of a PDP, i.e., the degree to which a user can rely on the estimate of the PD estimate, by comparing it to the true PD \(c_S(\lambda _S)\)  as defined in Eq. (REF ). More specifically, for every grid point \(\lambda _S^{(g)}\) , we compute the negative log-likelihood (NLL) of \(c_S(\lambda _S)\)  under the distribution of \(\hat{c}_S\left(\lambda _S\right)\)  pointwise for every grid point \(\lambda _S^{(g)}\) . The confidence of a PDP is illustrated by the width of its confidence bands \(\hat{m}_S\left(\lambda _S\right) \pm q_{1 - \alpha /2}\cdot \hat{s}_S\left(\lambda _S\right)\) , with \(q_{1 - \alpha /2}\)  denoting the \((1 - \alpha / 2)\) -quantile of a standard normal distribution. We measure the confidence by assessing \(\hat{s}_S(\lambda _S)\)  pointwise for every grid point. In particular, we consider the mean confidence (MC) across all grid points \(\frac{1}{G} \sum _{g = 1}^G \hat{s}\left(\lambda _S^{(g)}\right)\)  as well as the confidence at the grid point closest to \(\hat{\lambda }_S\)  abbreviated by OC, with \(\hat{\lambda }\)  being the best configuration evaluated by the optimizer.
To evaluate the performance of the confidence splitting, we report the above metrics on the sub-region that contains the best configuration evaluated by the optimizer, assuming that this region is of particular interest for a user of HPO.
PDPs are computed with regards to single features for \(G = 20\)  equidistant grid points and \(n = 1000\)  Monte Carlo samples.
