We have seen that over-parameterization helps explain why Gradient Descent can reach global minima even for highly non-convex optimization landscapes. Yet, in practice, GD is rarely used. Instead, mini-batch stochastic methods, such as SGD or Adam [1]} are employed almost exclusively.
In its simplest form, mini-batch SGD uses the following update rule:
\({\mathbf {w}}_{t+1}&={\mathbf {w}}_t -\eta \nabla \left(\frac{1}{m}\sum _{j=1}^m\ell (f({\mathbf {w}}_t,{\mathbf {x}}_{i_j}), y_{i_j})\right) \) 
