However, because the neural network is non-convex and non-continuous, we cannot find the optimal step size for each iteration. Therefore, in the stochastic gradient descent of the neural network, we empirically select a constant as the step size of each iteration. The constant step size has some shortcomings. If the constant step size is too large, the algorithm will oscillate or even fail to converge in the later stage. If the constant step size is too small, the algorithm will converge very slowly[1]}. Consequently, we use a learning rate with upper and lower bounds linearly decreasing with the number of iterations in Algorithm 2. As a result, the algorithm has a larger learning rate in the initial stage, speeds up the convergence speed, and has a smaller learning rate in the later stage, can converge stably.
<FIGURE>