More sophisticated methods include gradient-based methods such as [1]}, [2]}, [3]}, [4]} are applicable to continuous hyperparameters, but not to schedules which we consider. Bayesian optimization (BO) algorithms [5]}, [6]}, [7]}, [8]}, [9]}, [10]}, [11]}
tune hyperparameters by assuming a prior distribution of the loss function, and then keep updating this prior distribution based on the new observations. Each new observation is selected according to an acquisition function, which balances exploration and exploitation such that the new observation gives us a better result, or helps gain more information.
