Theorem 1 provides several interesting insights. RMSProp converges to the direction which is independent with the conditioner while AdaGrad does not. The limit direction of RMSProp is the same with gradient descent, which is the max-margin solution and indicates good generalization. It matches the empirical observations that RMSProp generalize better than AdaGrad on deep neural networks [1]}. The reason is the exponential weighted average strategy (i.e., \(m(t+1)=bm(t)+(1-b)\bar{\partial } \mathcal {L}^2\) , where \(\sqrt{m(t)+\varepsilon }=h(t)^{-1}\) ) in RMSProp makes the part \(m(t)\)  converge to zero, while in AdaGrad, the update rule is \(m(t+1)=m(t)+\bar{\partial } \mathcal {L}^2\) , which makes \(m(t)\)  keep growing and rely on the optimization process. Therefore, adaptive algorithms that adopt exponential weighted average strategy achieves tradeoff between optimization efficiency and generalization guarantee, i.e., it accelerates the training process at early stage of optimization by adaptively adjusting the learning rate, but it still converges to low-complexity model (i.e., max margin solution) because the conditioner converges to constant at later stage.
