A Survey of Optimization Methods from a Machine Learning Perspective <s> 1) First-Order Gradient Method in Deep Neural Networks: <s> Despite superior training outcomes, adaptive optimization methods such as Adam, Adagrad or RMSprop have been found to generalize poorly compared to Stochastic gradient descent (SGD). These methods tend to perform well in the initial portion of training but are outperformed by SGD at later stages of training. We investigate a hybrid strategy that begins training with an adaptive method and switches to SGD when appropriate. Concretely, we propose SWATS, a simple strategy which switches from Adam to SGD when a triggering condition is satisfied. The condition we propose relates to the projection of Adam steps on the gradient subspace. By design, the monitoring process for this condition adds very little overhead and does not increase the number of hyperparameters in the optimizer. We report experiments on several standard benchmarks such as: ResNet, SENet, DenseNet and PyramidNet for the CIFAR-10 and CIFAR-100 data sets, ResNet on the tiny-ImageNet data set and language modeling with recurrent networks on the PTB and WT2 data sets. The results show that our strategy is capable of closing the generalization gap between SGD and Adam on a majority of the tasks. <s> BIB001 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> 1) First-Order Gradient Method in Deep Neural Networks: <s> Several recently proposed stochastic optimization methods that have been successfully used in training deep networks such as RMSProp, Adam, Adadelta, Nadam are based on using gradient updates scaled by square roots of exponential moving averages of squared past gradients. In many applications, e.g. learning with large output spaces, it has been empirically observed that these algorithms fail to converge to an optimal solution (or a critical point in nonconvex settings). We show that one cause for such failures is the exponential moving average used in the algorithms. We provide an explicit example of a simple convex optimization setting where Adam does not converge to the optimal solution, and describe the precise problems with the previous analysis of Adam algorithm. Our analysis suggests that the convergence issues can be fixed by endowing such algorithms with `long-term memory' of past gradients, and propose new variants of the Adam algorithm which not only fix the convergence issues but often also lead to improved empirical performance. <s> BIB002
The stochastic gradient optimization method and its adaptive variants have been widely used in deep neural networks and have achieved good performance. SGD introduces the learning rate decay factor and AdaGrad accumulates all previous gradients so that their learning rates are continuously decreased and converge to zero. However, the learning rates of these two methods make the update slow in the later stage of optimization. AdaDelta, RMSProp, Adam and other methods use the exponential averaging to provide effective updates and simplify the calculation. These methods use exponential moving average to alleviate the problems caused by the rapid decay of the learning rate but limit the current learning rate to only relying on a few gradients BIB002 . Reddi et al. used a simple convex optimization example to demonstrate that the RMSProp and Adam algorithms could not converge BIB002 . Almost all the algorithms that rely on a fixed-size window of the past gradient will suffer from this problem, including AdaDelta and Nesterov-accelerated adaptive moment estimation (Nadam)  . It is better to rely on the long-term memory of past gradients rather than the exponential moving average of gradients to ensure convergence. A new of Adam BIB002 , called AmsGrad, uses a simple correction method to ensure the convergence of the model while preserving the original computational performance and advantages. Compared with the Adam method, the AmsGrad makes the following changes to the first-order moment estimation and the second-order moment estimation: where β 1t is a non-constant. The correction is operated in the second-order moment V t , makingV t monotonous.V t is substantially used in the iteration of the target function. The AmsGrad method takes the long-term memory of past gradients based on the Adam method, guarantees the convergence in the later stage, and works well in applications. Further, adjusting parameters β 1 , β 2 helps to converge to a certain extent. For example, β 1 can decay modestly as β 1t = β1 t , β 1t ≤ β 1 , for all t ∈ [T ]. β 2 can be set as , as in AdamNC algorithm BIB002 . Another idea of combining SGD and Adam was proposed for solving the non-convergence problem of adaptive gradient algorithm BIB001 . Adaptive algorithms, such as Adam, converge fast and are suitable for processing sparse data. SGD with momentum can converge to more accurate results. The combination of SGD and Adam develops the advantages of both methods. Specifically, it first trains with Adam to quickly drop, and then switches to SGD for precise optimization based on the previous parameters at an appropriate switch point. The strategy is named as switching from Adam to SGD (SWATS) BIB001 . There are two core problems in SWATS. One is when to switch from Adam to SGD, the other is how to adjust the learning rate after switching the optimization algorithm. The movement d Adam of the parameter at iteration t of the Adam is where η Adam is the learning rate of Adam BIB001 . The movement d SGD of the parameter at iteration t of the SGD is where η SGD is the learning rate of SGD and g t is the gradient of the current position BIB001 . The movement of SGD can be decomposed into the learning rates along Adam's direction and its orthogonal direction. If SGD is going to finish the trajectory but Adam has not finished due to momentum after selecting the optimization direction, walking along Adam's direction is a good choice for SWATS. At the same time, SWATS also adjusts its optimized trajectory by moving in the orthogonal direction. Let and derive solution where P roj Adam means the projection in the direction of Adam. To reduce noise, a moving average can be used to correct the estimate of the learning rate, where λ