Abstract Stochastic gradient descent (SGD) and Adam are commonly used to optimize deep neural networks, but choosing one usually means making tradeoffs between speed, accuracy and stability. Here we present an intuition for why the tradeoffs exist as well as a method for unifying the two in a continuous way. This makes it possible to control the way models are trained in much greater detail. We show that for default parameters, the new algorithm equals or outperforms SGD and Adam across a range of models for image classification tasks and outperforms SGD for language modeling tasks. One of the most common methods of training neural networks is stochastic gradient descent (SGD) (Bottou et al. (2016) ). SGD has strong theoretical guarantees, including convergence in locally non-convex optimization problems (Lee et al. (2016) ). It also shows improved generalization and stability when compared to other optimization algorithms (Smith & Le (2018) ). There have been various efforts in improving the speed and generalization of SGD. One popular modification is to use an adaptive gradient (Duchi et al. (2011) ), which scales the gradient step size to be larger in directions with consistently small gradients. Adam, an implementation that combines SGD with momentum and an adaptive step size inversely proportional to the RMS gradient, has been particularly successful at speeding up training and solving particular problems (Kingma & Ba (2014) ). However, at other problems it pays a penalty in worse generalization (Wilson et al. (2017) ; Keskar & Socher (2017) ), and it requires additional modifications to achieve a convergence guarantee (Reddi et al. (2018) ; Li & Orabona (2018) ). Here we develop an intuition for adaptive gradient methods that allows us to unify Adam with SGD in a natural way. The new optimizer, SoftAdam, descends in a direction that mixes the SGD with Adam update steps. As such, it should be able to achieve equal or better optimization results across a variety of problems. In this paper, we have motivated and demonstrated a new optimization algorithm that naturally unifies SGD and Adam. We have focused our empirical results on the default hyper-parameter setting, η = 1, and predetermined learning schedules. With these parameters, the algorithm was shown to produce optimization that is better than or equal to SGD and Adam on image classification tasks. It also performed significantly better than SGD on language modeling tasks. Together with finding the optimal values for η, we expect a better understanding of the learning schedule to bring light to the way in which the adaptive gradient methods improve convergence. SoftAdam now also makes it possible to create a learning schedule on η, which may be another fruitful avenue of research, expanding on the work of Ward et al. (2018) . Better understanding of how adaptive gradients improve the convergence of practical machine learning models during training will enable larger models to be trained to more accurately in less time. This paper provides a useful intuition for how that occurs and provides a new algorithm that can be used to improve performance across a diverse set of problems. # S t a t e i n i t i a l i z a t i o n i f l e n ( s t a t e ) == 0 : s t a t e [ " s t e p " ] = 0 # E x p o n e n t i a l moving a v e r a g e o f g r a d i e n t v a l u e s s t a t e [ " e x p a v g " ] = t o r c h . z e r o s l i k e ( p . d a t a ) # E x p o n e n t i a l moving a v e r a g e o f # s q u a r e d g r a d i e n t v a l u e s s t a t e [ " e x p a v g s q " ] = t o r c h . z e r o s l i k e ( p . d a t a ) e x p a v g , e x p a v g s q = ( s t a t e [ " e x p a v g " ] , s t a t e [ " e x p a v g s q " ] , ) b e t a 1 , b e t a 2 = g r o u p [ " b e t a s " ] s t a t e [ " s t e p " ] += 1 b e t a 2 h a t = min ( b e t a 2 , 1 . 0 − 1 . 0 / ( s t a t e [ " s t e p " ] ) ) r b e t a = ( 1 − b e t a 2 ) / ( 1 − b e t a 2 h a t ) e t a h a t 2 = ( g r o u p [ " e t a " ] * g r o u p [ " e t a " ] * r b e t a ) # Decay t h e f i r s t and s e c o n d moment w i t h t h e # r u n n i n g a v e r a g e c o e f f i c i e n t e x p a v g . mul ( b e t a 1 ) . a d d r e t u r n l o s s
An algorithm for unifying SGD and Adam and empirical study of its performance
{"src": "scitldr-AIC_train", "paper_id": "Skgfr1rYDH"}