We evaluate how well our and other performance estimation strategies can predict the actual accuracy of a candidate model in terms of RMSE, MAE and Pearson's correlation coefficient, Spearman's Rho, and Kendall Tau. Based on Pearson and Spearman, [1]} found that the best methods were Early Stopping (on validation accuracy), GCN [2]}, Jacobian Covariance [3]}, LcSVR [4]}, SoTL-E [5]}, SemiNAS [6]} and XGBoost [7]}. However, not all of these methods could be compared to ours in a fair way. For instance, Early Stopping is not model-based and could hence not be compared in terms of fit time. Others, such as GCN, far exceeded our resource limitations, which made the comparison infeasible as well as less informative given our focus on efficiency and previously published benchmarks. Hence, to give a fair overview of competitors we chose to benchmark against NAO, SemiNAS, XGBoost and BANANAS [8]} and an MLP.
