
From the representation (REF ), it can be seen that, if \(\hat{\beta }\)  is well-defined, then \(\hat{\beta }\)  interpolates the data in the sense that \(\langle X_i, \hat{\beta }\rangle \)  and \(Y_i\)  have matching signs for all \(i\) . Similarly, neural networks and random forests are typically massively overparameterized and trained until they interpolate the data. Empirically, it has been shown that this can lead to smaller test errors compared to algorithms with a smaller number of parameters [1]}, [2]}. Statistical learning theory based on empirical risk minimization techniques and entropy bounds can not explain these empirical findings and a mathematical understanding of this phenomenon has only began to form in recent years. The prevalent explanation so far is that, similar as in (REF ), these algorithms approximate max-margin solutions [3]}, [4]}, [5]}. As in (REF ), an algorithm that maximises a margin is equivalent to a minimum-norm-interpolator. It is then argued that this leads to implicit regularization and hence a good fit.
