Before introducing SGD and its derivatives, we first discuss their classical versions.
As a first-order optimization method, gradient descent (GD) is arguably the simplest one.
Unlike second-order methods, such as Newton-Raphson algorithm, a first-order method requires no computation of the Hessian matrices.
In fact, a standard GD algorithm only requires the gradients of the objective function and a learning rate.
Therefore, it is computationally more feasible for sophisticated models with massive datasets. Furthermore, the cost in computer memory for storing Hessian matrices can be significantly reduced.
Unfortunately, there is no free lunch. The tradeoff for GD methods' excellent practical feasibility is slow convergence. In other words, the convergence rate of a GD algorithm is usually much slower than that of a Newton-Raphson algorithm, if the Newton-Raphson algorithm can be implemented. This is particularly true for ill-conditioned problems [1]}. In this case, feature directions with tiny variability are likely to be ignored by a standard GD algorithm. To partially fix this problem, [2]} proposed a heavy-ball method to accelerate the standard GD algorithm. Specifically, it uses the information from not only the current step but also the previous step (i.e., the momentum). As a consequence, the search direction can be slightly modified. With this slight but novel modification, the convergence rate of gradient descent with momentum (GDM) could be much improved compared to the standard GD algorithm.
