On the other hand, adaptive algorithms such as AdaGrad [1]}, RMSProp [2]}, and Adam [3]} have been in spotlight these years. These algorithms are proposed to improve the convergence rate of GD (or SGD) by
using second-order moments of historical gradients and have been widely applied in deep learning [4]}. Despite the rapid convergence of adaptive methods, numerous works have provided empirical evidence that adaptive methods may suffer from poor generalization performance [5]}, [6]}. Specifically, it has been observed that first-order method such as SGD can outperform adaptive methods on the test set, even when the adaptive methods achieve the same or lower training loss.
Several works try to improve the performance of adaptive optimization algorithms such as AdamW [7]}, AdaBound [6]}, AdaBelief [9]}. However, there is little theoretical analysis for generalization of adaptive algorithms. These observations and the research for GD motivate us to study the implicit bias for adaptive algorithms.
