At a first glance, from a classical point of view, it appears that GD should be preferable to SGD. In a standard convex setting GD converges at an exponential (referred as linear in the optimization literature) rate, where the loss function decreases exponentially with the number of iterations.
In contrast, while SGD requires a factor of \(\frac{n}{m}\)  less computation than GD per iteration, it converges at a far slower sublinear rate (see [1]} for a review), with the loss function decreasing
proportionally to the inverse of the number of iterations.
Variance reduction techniques [2]}, [3]}, [4]} can close the gap theoretically but are rarely used in practice.
