The key factor for the success of adaptive algorithms is to design better conditioners of the gradient. There are two popular kinds of conditioners: one is the simple average of the squared values of past gradients that is used in AdaGrad, and the other is the exponential moving average that is used in RMSProp, Adam, etc. In this paper, we aim to study the influence of these two kinds of conditioners on convergent direction of parameters on deep neural networks. Specifically, we work on the homogeneous neural networks with separable data under logistic regression setting. Unlike the squared loss, the logistic doss does not admit a finite global minimizer on separable data: the iterates will diverge in order to drive the loss to zero. Therefore, researchers use the asymptotic direction of algorithms (i.e., \(\lim _{t\rightarrow \infty } \frac{w_t}{\Vert w_t\Vert _2}\) ) for logistic loss instead of characterizing the exact limit of parameters [1]}, [2]}.
