After showing the details of these traditional adaptive gradient methods and Adam-type methods, we introduce their convergence properties as well as their further variants. [1]} shows that Adam does not converge in some settings where large gradient information is rarely encountered and it will die out quickly because of the “short memory” property of the exponential moving average. However, under some conditions, the convergence proofs of adaptive gradient methods have been obtained. [2]} proved the convergence rate of RMSprop and Adam when using deterministic gradients instead of stochastic gradients. [3]} analyzed the convergence rate of AdaGrad under both convex and non-convex settings. All the papers above provide theoretical guarantee for the convergence of different types of adaptive gradient descent. After that, [4]} extends Adam to a broader class of Adam-type algorithms and provides its convergence analysis for non-convex optimization problems. In order to combine the fast convergence of adaptive methods and better generalization with momentum based methods, a number of new algorithms are proposed, such as SC-AdaGrad / SC-RMSprop [5]}, AdamW [6]}, AdaBound [7]} etc..
