Yet, even in the 1990's it had become clear that successes of Adaboost [1]} and neural networks were difficult to explain from the SRM or bias-variance trade-off paradigms. Leo Breiman, a prominent statistician, in his note [2]} from 1995 posed the question “Why don't heavily parameterized neural networks overfit the data?”.
In particular, it was observed that increasing complexity of classifiers (capacity of \({\mathcal {H}}\) ) in boosting did not necessarily lead to the expected drop of performance due to over-fitting. Why did the powerful mathematical formalism of uniform laws of large numbers fail to explain the observed evidenceThis question appears as a refrain throughout the history of Machine Learning and, perhaps, other domains.?
