As we will see, interpolation, the idea of fitting the training data exactly, and its sibling over-parameterization, having sufficiently many parameters to satisfy the constraints corresponding to fitting the data, taken together provide a perspective on some of the more surprising aspects of neural networks and other inferential problems.
It is interesting to point out that
interpolating noisy data is a deeply uncomfortable and counter-intuitive concept to statistics, both theoretical and applied, as it is traditionally concerned with over-fitting the data. For example, in a book on non-parametric statistics [1]}(page 21) the authors dismiss a certain procedure on the grounds that it “may lead to a function which interpolates the data and hence is not a reasonable estimate”. Similarly, a popular reference [2]}(page 194) suggests that “a model with zero training error is overfit to the training data and will typically generalize poorly”.
