Although the optimal numerical convergence can be guaranteed by (a) and (b) under different circumstances, the stable solution is not necessarily optimal in statistical efficiency.
For most SGD-related methods, the decaying learning rate \(\alpha \)  is typically needed to improve statistical efficiency [1]}, [2]}.
In this case, from Theorem REF  (c.1) we know that the convergence factor in this case tends to 1 as \( \alpha \rightarrow 0\) , which represents a very slow convergence rate.
Moreover, from Theorem REF  (c.2) we find that by choosing an appropriate \( \gamma \) , the convergence factor of the FMGDM could be strictly smaller than that of a standard GD algorithm (\( \gamma =0 \) ).
This finding suggests that the momentum term can indeed accelerate the numerical convergence.
In addition, Theorem REF  (c.2) discovers a very interesting and novel interactive relationship between the learning rate \( \alpha \)  and the momentum parameter \( \gamma \) . Specifically, it suggests that \( \gamma \)  should approach 1 as \( \alpha \rightarrow 0 \) . This result theoretically explains why practitioners often like to specify \( \gamma \)  to be a value very close to 1 to strike a balance between both numerical and statistical convergence [3]}.
