First, Theorem REF  and Theorem REF  shows that RMSProp implicitly bias the solution to ones with maximized surrogate margin, while AdaGrad may drive the parameters to a different direction. The corresponding optimization problem of AdaGrad has a reliance on \(h_{\infty }\) , which is shown to be sensitive to the optimization path before convergence. Because the normalized margin is used as a complexity norm in generalization literature (i.e., larger normalized margin indicating better generalization performance) [1]}, we conclude that RMSProp generalizes better than AdaGrad. This conclusion matches the empirical observations [2]}.
