As it turns out, interpolation can explain the surprising effectiveness of plain SGD compared to GD and other non-stochastic methodsNote that the analysis is for the convex interpolated setting. While bounds for convergence under the PL* condition are available [1]}, they do not appear to be tight in terms of the step size and hence do not show an unambiguous advantage over GD. However, empirical evidence suggests that analogous results indeed hold in practice for neural networks.
