At present, researchers have also proposed a large number of training algorithms, such as the classic stochastic gradient descent[1]}. The stochastic gradient descent has low computational complexity due to its simplicity, but the training process completely depends on the current gradient. This algorithm has low robustness and the training process is unstable. Polyak et al.[2]} proposed a stochastic descent with momentum, but it still used a constant learning rate to scale the gradient in all directions. Duchi et al.[3]} proposed the adaptive gradient descent AdaGrad using the second moment as the adaptive learning rate. Subsequently, researchers improved AdaGrad and proposed RMSProp[4]}, AdaDelta[5]} and so on. Kingma et al.[6]} combined the advantages of AdaGrad and RMSProp, and proposed the Adam algorithm. Adam algorithm has the characteristics of fast training speed, good robustness and stability, and is widely used in various neural network. Reddi et al.[7]} gave a convex optimization problem to show that the Adam algorithm cannot converge. The reason is that the Adam algorithm has short-term memory for the second moment. In order to correct this defect, the AmsGrad algorithm was proposed.
