where \( \alpha >0\)  is the learning rate and \(\gamma >0 \)  is the momentum parameter. [1]} showed that, by choosing appropriate tuning parameters \( \alpha \)  and \( \gamma \) , the optimal convergence factor of the momentum method could be achieved as \( \rho = (\sqrt{\kappa } - 1) / (\sqrt{\kappa } + 1) \) . In contrast, that of a standard GD algorithm is \( \rho = (\kappa - 1)/ (\kappa +1) \) , which is strictly larger than that of the GD with momentum (GDM) method as long as \( \kappa >1 \) . This suggests that the optimal convergence rate of a GDM algorithm should be faster than that of a standard GD algorithm.
