SGDR: Stochastic Gradient Descent with Restarts
Restart techniques are common in gradient free optimization to deal with multimodal functions. Partial restarts are also gaining popularity in gradient based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill conditioned functions. In this paper, we propose a simple restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on CIFAR 10 and CIFAR 100 datasets where we demonstrate new state of the art results below 4\ and 19\ respectively. Our source code is available at this https URL
ArXiv