The adaptive gradient descent has greatly accelerated the training speed of neural networks. In particular, the Adam algorithm is widely used and even some models use it as the default training algorithm[1]}. However, some researchers pointed out that the generalization ability of adaptive gradient descent is not as good as stochastic gradient descent in some aspects[2]}. Reddi et al.[3]} proposed the AmsGrad algorithm with long-term memory for second moment in response to the defect of Adam's theory. Luo[4]} et al. conjectured and verified that the performance improvement of the AmsGrad algorithm is to avoid the extremely large learning rate. The transition from adaptive gradient descent to stochastic gradient descent was realized by dynamically clipping the boundary of adaptive learning rate. This avoids the influence of the extremely small and large adaptive learning rate.
