In this section, we mainly discuss the influence of constant learning rate and extreme adaptive learning rate on stochastic gradient descent and adaptive gradient descent. In optimization theory, the gradient descent is an important method to solve unconstrained optimization problems. When the objective function to be solved is convex and continuously differentiable, we can use the steepest descent method to find the optimal solution through linear search. Through the iterative process of the steepest descent method, we can know that there is a larger step size at the beginning, and the step size near the optimal solution is getting smaller and smaller[1]}. That is to say, in the initial stage, a larger step size can have a faster convergence speed, and a smaller step size when approaching the optimal solution is to avoid oscillations.
