
First observe that an interpolating learning algorithm
\({\mathcal {A}}\)  selects a specific predictor \(f_{\mathcal {A}}\in {\mathcal {S}}\) . Thus we are faced with the issue of the inductive bias: why do solutions, such as those obtained by neural networks and kernel machines, generalize, while other possible solutions do notThe existence of non-generalizing solutions is immediately clear by considering over-parameterized linear predictors. Many linear functions fit the data – most of them generalize poorly..
Notice that this question cannot be answered through the training data alone, as any \(f \in {\mathcal {S}}\)  fits data equally wellWe note that inductive biases are present in any inverse problem. Interpolation simply isolates this issue..
While no conclusive recipe for selecting the optimal \(f \in {\mathcal {S}}\)  yet exists, it can be posited that an appropriate notion of functional smoothness plays a key role in that choice. As argued in [1]}, the idea of maximizing functional smoothness subject to interpolating the data represents a very pure form of the Occam's razor (cf. [2]}, [3]}). Usually stated as
