Adaptive gradient methods, especially the Adam algorithm [1]} are nowadays the standard for training deep neural networks. This is because they are much easier to handle (less sensitive to weight initialization and hyperparameters) compared with mini-batch stochastic gradient descent [2]} (SGD). This can be attributed to two key properties of the Adam algorithm: adaptive learning rate and momentum term. Via the adaptive learning rate, the algorithm computes an individual learning rate for each parameter of the neural network model - in contrast to SGD which keeps a fixed learning rate for all parameters of the model. This is especially helpful for neural networks with many different types of layers, with the parameters in each layer typically having a different "typical" value range. On the other side, via the momentum term it can keep progress even when encountering regions of the loss function landscape with high curvature, which resemble long and narrow ravines (mathematically speaking, these are regions where the Hessian matrix has a high condition number).
