For kernel machines with standard Laplacian and Gaussian kernels, a setting where both early stopping and exact solutions can be readily computed, early stopping seems to provide at best a modest improvement to generalization performance [1]}.
Yet, even for kernel machines, computational efficiency of training on larger datasets seems to require iterative methods similar to SGD [2]}, [3]}, thus making early stopping a computational necessity.
