It can be seen that the scaling operation of \(\displaystyle {\hat{v}_t}\)  is similar to the linear weighted moving average of the adaptive learning rate and the decreasing learning rate. But the weight coefficient here is an exponential function that decreases with the number of iterations, which scales the adaptive learning rate \(\displaystyle a/\sqrt{{v_t}} \) . \(\displaystyle \gamma \)  is called the transition factor. It controls the moment when the algorithm transforms from adaptive gradient descent to stochastic gradient descent. In fact, when \(\displaystyle \gamma \)  is large enough, Algorithm 2 is a stochastic gradient descent. When \(\displaystyle \gamma \)  is close enough to 1, Algorithm 2 is an adaptive gradient descent. The DSTAda algorithm is close to the Adam algorithm in the initial stage, and close to the stochastic gradient descent in the later stage. It combines the advantages of fast training speed of adaptive gradient descent and high training accuracy of Stochastic gradient descent. According to [1]}, [2]}, [3]}, we mainly give the following important conclusions for the DSTAda algorithm.
