To avoid confusion, it is important to emphasize that interpolation is not necessary for good generalization. In certain models (e.g., [1]}), introducing some regularization is provably preferable to fitting the data exactly. In practice, early stopping is typically used for training neural networks. It prevents the optimization process from full convergence and acts as a type of regularization [2]}.
What is remarkable is that interpolating predictors often provide strong generalization performance, comparable to the best possible predictors.
Furthermore, the best practice of modern deep learning is arguably much closer to interpolation than to the classical regimes (when training and testing losses match). For example in his 2017 tutorial on deep learning [3]} Ruslan Salakhutdinov stated that “The best way to solve the problem from practical standpoint is you build a very big system \(\ldots \)  basically you want to make sure you hit the zero training error”. While more tuning is typically needed for best performance, these “overfitted” systems already work well [4]}.
Indeed, it appears that the largest technologically feasible networks are consistently preferable for best performance. For example, in 2016 the largest neural networks had fewer than \(10^9\)  trainable parameters [5]}, the current (2021) state-of-the-art Switch Transformers [6]} have over \(10^{12}\)  weights, over three orders of magnitude growth in under five years!
