Technically, although we adopt the framework of [1]}, the techniques for vanilla GD can no longer be applied to adaptive algorithms. We highlight our technical contributions as: (1) we construct adaptive gradient flow, which is novel and non trivial. With the adaptive gradient flow, the analysis of convergent direction is transformed from original parameter space to a normalized parameter space. The adaptive gradient flow can be applied into analysis of other adaptive algorithms. (2)
Previous works [2]}, [1]} prove for vanilla GD, smoothed margin defined as \(\frac{\log \frac{1}{\mathcal {L}(w)}}{\Vert w\Vert ^L}\)  monotonously increases, which however, in adaptive methods, no longer holds. Instead, we define the surrogate margin which take the conditioner into account, and prove the surrogate margin (i) is lower bounded; (ii) has the same limit with the normalized margin.
