Keskar et al.[1]} improved the training speed and generalization ability of the neural network through the manual transition from the adaptive gradient descent to the stochastic gradient descent. The transition is performed at a certain moment in the training process. The author gives a condition for the transition point to be \(\displaystyle  k > 1\)  and \(\displaystyle \left| {{\lambda _k}/\left( {1 - \beta _2^k} \right) - {\gamma _k}} \right| < \varepsilon \) . However, choosing a certain transition point for "hard" transition may not be the best[1]}. Luo et al.[3]} studied the influence of extreme learning rate on this basis. An adaptive gradient descent with dynamic boundary learning rate(AdaBound) was proposed, which realized the smooth transition from adaptive gradient descent to stochastic gradient descent. However, two boundary functions need to be manually set in AdaBound, and the performance of the algorithm has a large dependence on the boundary functions[4]}. In some aspects, the AdaBound algorithm does not have a good performance[5]}. Therefore, the research on the combination of adaptive gradient descent and stochastic gradient descent still needs some effort. How to achieve a better transition while retaining the advantages of the fast convergence speed of the adaptive gradient descent and the good generalization ability of the stochastic gradient descent is very important.
