Hyperparameter tuning is arguably the most important ingredient for obtaining state of art performance in deep networks.   We focus on hyperparameters  that are related to the optimization algorithm, e.g.  learning rates, which have a large impact on the training speed and the resulting accuracy. Typically, fixed learning rate schedules are employed during training. We propose Hyperdyn a dynamic hyperparameter optimization method that selects new learning rates on the fly at the end of each epoch. Our explore-exploit   framework  combines Bayesian optimization (BO) with a rejection strategy, based on a simple probabilistic wait and watch test.    We obtain state of  art accuracy results on CIFAR and Imagenet datasets, but with significantly faster training, when compared with the best manually tuned networks. Hyperparameter tuning is arguably the most important ingredient for obtaining state of art performance in deep neural networks. Currently, most networks are manually tuned after extensive trial and error, and this is derisively referred to as graduate student descent. Hyperparameter optimization (HPO), on the other hand, attempts to automate the entire process and remove the need for human intervention. Previous works on HPO propose various strategies to either adaptively select good configurations BID12 or to speed up configuration evaluations BID7 .One drawback behind existing HPO frameworks is that they do not distinguish between different kinds of hyperparameters and treat them all uniformly. Broadly there are two categories of hyperparameters: those that are fixed throughout the training process and those that need to be varied during training. The former class is mostly structural (e.g. network depth and width), while the latter is mostly related to the optimization algorithm (e.g. learning rates, regularization). The two classes of hyperparameters have very different behaviors. For structural hyperparameters, online evaluation is not possible, and we need to wait for the entire training process to be completed, which is very expensive. On the other hand , for time-varying hyperparameters, it is possible to select parameters on the fly without waiting for training to finish. In this work we keep the structural hyperparameters fixed and focus on optimizing the time-varying hyperparameters to improve efficiency, stability, and accuracy of the training process. In this work we introduced a general framework for a certain class of hyperparameter optimization. The algorithm here has the advantage that it is fast, flexible -can be used on any objective function and training method, and stable on larger batch sizes. We demonstrated that our proposed optimizer is faster and at least as accurate as SGD (with momentum) or Adam. We also show that Hyperdyn is stable for larger batch sizes on Imagenet (achieves acceptable accuracy with 4x speed). We demonstrate how too much exploration can be detrimental to generalization accuracy of a training process, and propose a probabilistic "wait-and-watch" strategy.Currently we do not parallelize Hyperdyn ; however, computing Validation Accuracy on the suggestion from One Step BO can be easily parallelized. At each epoch we make only suggestion and the validation accuracy on this suggestion can be computed independently of the current hyperparameter setting. In the general case, when we make multiple suggestions we can parallelize in a similar fashion to BID12 . We also observe that epoch-based BO in FIG1 outperforms Hyperdyn in the initial epochs. One future direction maybe to have time varying temperature in Check Accuracy Trend based on epoch. We can also exploit the temporal gains obtained by using a backtrack algorithm at the later stages of the algorithm -at a stage when accuracy is sufficiently high but more sophisticated tuning is required to get the best error rates.
Bayesian optimization based online hyperparameter optimization.
{"src": "scitldr-AIC_train", "paper_id": "HJtPtdqQG"}