The development of gradient descent improves the performance of neural network to a certain extent. However, both the stochastic gradient descent and the adaptive gradient descent have their own advantages and disadvantages. For example, although the stochastic gradient descent has slower training speed, unstable training, and easy shock in the training process, it often has better generalization ability in the later stage of training. The adaptive gradient descent has a faster speed in the early stage, but it is easy to fall into the local optimum and poor generalization ability in the later stage of training. Therefore, the combination of adaptive gradient descent and stochastic gradient descent has become a research hotspot[1]}, [2]}, [3]}, [4]}, [5]}. This can make the neural network have a faster speed in the early stage, and at the same time have a better generalization ability in the later stage of training.
