where \(\bf X\)  is the data matrix, \({\mathbf {y}}\)  is the vector of labels and \({\bf X}^\dagger \)  is the Moore-Penrose (pseudo-)inverseIf \({\mathbf {X}}{\mathbf {X}}^T\)  is invertible, as is usually the case in over-parameterized settings, \({\mathbf {X}}^\dagger = {\mathbf {X}}^T({\mathbf {X}}{\mathbf {X}}^T)^{-1}\) . In contrast, if \({\mathbf {X}}^T{\mathbf {X}}\)  is invertible (under the classical under-parameterized setting), \({\mathbf {X}}^\dagger =({\mathbf {X}}^T{\mathbf {X}})^{-1}{\mathbf {X}}^T\) . Note that both \({\mathbf {X}}{\mathbf {X}}^T\)  and \({\mathbf {X}}^T{\mathbf {X}}\)  matrices cannot be invertible unless \(X\)  is a square matrix, which occurs at the interpolation threshold..
Linear regression for models of the type in Eq. REF  is no doubt the oldestOriginally introduced by Gauss and, possibly later, Legendre! See [1]}. and best studied family of statistical methods. Yet, strikingly, predictors such as those in Eq. REF , have historically been mostly overlooked, at least for noisy data.
Indeed, a classical prescription is to regularize the predictor by, e.g., adding a “ridge” \(\lambda I\)  to obtain a non-interpolating predictor.


The reluctance to overfit inhibited exploration of a range of settings where \(y({\mathbf {x}})=\langle {\beta }_\text{int},{\mathbf {x}}\rangle \)  provided optimal or near-optimal predictions. Very recently, these “harmless interpolation” [2]} or “benign over-fitting” [3]} regimes have become a very active direction of research, a development inspired by efforts to understand deep learning. In particular, the work [3]} provided a spectral characterization of models exhibiting this behavior. In addition to the aforementioned papers, some of the first work toward understanding “benign overfitting” and double descent under various linear settings include [5]}, [6]}, [7]}, [8]}.
Importantly, they demonstrate that when the number of parameters varies, even for linear models over-parametrized predictors are sometimes preferable to any “classical” under-parameterized model.
