A Survey of Optimization Methods from a Machine Learning Perspective <s> 4) Adaptive Learning Rate Method: <s> We present and compare learning rate schedules for stochastic gradient descent, a general algorithm which includes LMS, on-line backpropagation and k-means clustering as special cases. We introduce "search-then-converge" type schedules which outperform the classical constant and "running average" (1/t) schedules both in speed of convergence and quality of solution. <s> BIB001 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> 4) Adaptive Learning Rate Method: <s> We present a new family of subgradient methods that dynamically incorporate knowledge of the geometry of the data observed in earlier iterations to perform more informative gradient-based learning. Metaphorically, the adaptation allows us to find needles in haystacks in the form of very predictive but rarely seen features. Our paradigm stems from recent advances in stochastic optimization and online learning which employ proximal functions to control the gradient steps of the algorithm. We describe and analyze an apparatus for adaptively modifying the proximal function, which significantly simplifies setting a learning rate and results in regret guarantees that are provably as good as the best proximal function that can be chosen in hindsight. We give several efficient algorithms for empirical risk minimization problems with common and important regularization functions and domain constraints. We experimentally study our theoretical analysis and show that adaptive subgradient methods outperform state-of-the-art, yet non-adaptive, subgradient algorithms. <s> BIB002 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> 4) Adaptive Learning Rate Method: <s> We present a novel per-dimension learning rate method for gradient descent called ADADELTA. The method dynamically adapts over time using only first order information and has minimal computational overhead beyond vanilla stochastic gradient descent. The method requires no manual tuning of a learning rate and appears robust to noisy gradient information, different model architecture choices, various data modalities and selection of hyperparameters. We show promising results compared to other methods on the MNIST digit classification task using a single machine and on a large scale voice dataset in a distributed cluster environment. <s> BIB003 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> 4) Adaptive Learning Rate Method: <s> We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm. <s> BIB004 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> 4) Adaptive Learning Rate Method: <s> Recurrent neural network (RNN) and long short-term memory (LSTM) have achieved great success in processing sequential multimedia data and yielded the state-of-the-art results in speech recognition, digital signal processing, video processing, and text data analysis. In this paper, we propose a novel action recognition method by processing the video data using convolutional neural network (CNN) and deep bidirectional LSTM (DB-LSTM) network. First, deep features are extracted from every sixth frame of the videos, which helps reduce the redundancy and complexity. Next, the sequential information among frame features is learnt using DB-LSTM network, where multiple layers are stacked together in both forward pass and backward pass of DB-LSTM to increase its depth. The proposed method is capable of learning long term sequences and can process lengthy videos by analyzing features for a certain time interval. Experimental results show significant improvements in action recognition using the proposed method on three benchmark data sets including UCF-101, YouTube 11 Actions, and HMDB51 compared with the state-of-the-art action recognition methods. <s> BIB005
The manually regulated learning rate greatly influences the effect of the SGD method. It is a tricky problem for setting an appropriate value of the learning rate BIB002 , BIB004 , BIB001 . Some adaptive methods were proposed to adjust the learning rate automatically. These methods are free of parameter adjustment, fast to converge, and often achieving not bad results. They are widely used in deep neural networks to deal with optimization problems. The most straightforward improvement to SGD is AdaGrad BIB002 . AdaGrad adjusts the learning rate dynamically based on the historical gradient in some previous iterations. Different parameters {θ j } have different learning rates. The update formulae are as follows: where g t j is the gradient of parameter θ j at iteration t, V t j is the accumulate historical gradient of parameter θ j at iteration t, and θ t j is the value of parameter θ j at iteration t. The difference between AdaGrad and gradient descent is that during the parameter update process, the learning rate is no longer fixed, but is computed by using all the historical gradients accumulated up to this iteration. One main benefit of AdaGrad is that it eliminates the need to tune the learning rate manually. Most implementations use a default value of 0.01 for η in BIB005 . Although AdaGrad adaptively adjusts the learning rate, it still has two issues. 1) The algorithm still needs to set the global learning rate η manually. 2) As the training time increases, the accumulated gradient will become larger and larger, making the learning rate tend to zero, resulting in ineffective parameter update. AdaGrad was further improved to AdaDelta BIB003 and RMSProp  for solving the problem that the learning rate will eventually go to zero. The idea is to consider not accumulating all historical gradients, but focusing only on the descending gradient in a window over a period, and using the exponential moving average to calculate the second-order cumulative momentum, where β is the exponential decay parameter. Both RMSProp and AdaDelta have been developed independently around the same time, stemming from the need to resolve the radically diminishing learning rates of AdaGrad. Adaptive moment estimation (Adam) BIB004 is another advanced SGD method, which introduces an adaptive learning rate for each parameter. It combines the adaptive learning rate and momentum methods. In addition to storing an exponentially decaying average of past squared gradients V t , like AdaDelta and RMSProp, Adam also keeps an exponentially decaying average of past gradients m t , similar to the momentum method: where β 1 and β 2 are exponential decay rates for the moments estimate. The final update formula for the parameter θ j is as The default values of β 1 , β 2 , and ǫ are suggested to set to 0.9, 0.999, and 10 −8 , respectively. Adam works well in practice and compares favorably to other adaptive learning rate algorithms.