As one of the most popular optimizers in supervised deep learning tasks like natural language processing [1]} as well as the main workhorse of generative adversarial network training [2]}, Adam-type methods are widely used because of their minimal need for learning rate tuning and their coordinate-wise adaptivity on local geometry. Starting from AdaGrad [3]}, adaptive gradient methods have evolved into a variety of different Adam-type algorithms, such as Adam [4]}, RMSprop, AMSGrad [5]} and AdaDelta [6]}. In supervised deep learning, adaptive gradient methods and Adam-type algorithms play important roles. Especially in the field of NLP (natural language processing), Adam-type algorithms are the goto optimizer for NLP tasks. Multiple NLP experiments show that sparse Adam outperforms other non-adaptive algorithms like Stochastic Gradient Descent (SGD) not only on the solution performance and the loss curvature smoothness, but also on both the training and testing error's convergence rates. It's worth mentioned that the most popular pre-training language model BERT [7]} also uses Adam as its optimizer, which shows the power of Adam-type algorithms.
