In order to prove the Theorem 1, we first propose adaptive gradient flow, which is a unified framework to deal with adaptive gradients. It transforms original parameter space to a parameter space normalized by the conditioners. Second, in the normalized parameter space, we construct surrogate margin for the adaptive algorithms. The surrogate margin is defined as the ratio of negative logarithmic loss normalized by parameter norm. We prove that surrogate margin is lower bounded, and then the increasing rate of parameter norm can then be bounded by the decreasing rate of loss [1]}, which together drive the loss to zero. At last, we can prove any limit direction of the normalized parameter flow is the KKT point of the margin maximization problem in normalized parameter space. Moreover, we prove the convergent direction is unique if the neural network is definable [2]}.
