Theoretical Evidence of Generalization of Adaptive Algorithms. [1]} constructed a linear regression task where AdaGrad, RMSProp, and ADAM converge to a solution that incorrectly classifies new data with probability arbitrarily close to half and SGD finds a solution with zero error on new data. [2]} modeled the distribution of stochastic noise in Adam, and they showed that SGD tends to converge to flatter local minima.
Another viewpoint is to study the implicit bias of adaptive optimization algorithms. To the best of our knowledge, the only work on the bias of adaptive algorithms is [3]}, which proves AdaGrad has different asymptotic direction with GD on linear logistic regression. In this paper, we study the asymptotic direction of adaptive optimization algorithms on deep neural networks which requires different techniques due to high non-convex of deep networks.
