In this section, we review two popular gradient descent methods, namely, the basic gradient descent method and the Adam optimizer. In each iteration, the basic gradient descent method uses only the current gradient for finding the descent direction \(\Delta \mathbf {w}\) . On the other hand, the Adam optimizer uses the current as well as all the past gradients for finding \(\Delta \mathbf {w}\) , in such a way that the algorithm moves with momentum that can provide acceleration and can help in bypassing saddle points and local minima of the objective function. See, for example, [1]},
for more details on gradient descent methods.
