Max-Margin Works while Large Margin Fails: Generalization without Uniform Convergence
A major challenge in modern machine learning is theoretically understanding
the generalization properties of overparameterized models. Many existing tools
rely on \em uniform convergence \em (UC), a property that, when it holds,
guarantees that the test loss will be close to the training loss, uniformly
over a class of candidate models. Nagarajan and Kolter (2019) show that in
certain simple linear and neural-network settings, any uniform convergence
bound will be vacuous, leaving open the question of how to prove generalization
in settings where UC fails. Our main contribution is proving novel
generalization bounds in two such settings, one linear, and one non-linear. We
study the linear classification setting of Nagarajan and Kolter, and a
quadratic ground truth function learned via a two-layer neural network in the
non-linear regime. We prove a new type of margin bound showing that above a
certain signal-to-noise threshold, any near-max-margin classifier will achieve
almost no test loss in these two settings. Our results show that
near-max-margin is important: while any model that achieves at least a $(1 -
\epsilon)$-fraction of the max-margin generalizes well, a classifier achieving
half of the max-margin may fail terribly. We additionally strengthen the UC
impossibility results of Nagarajan and Kolter, proving that \em one-sided \em
UC bounds and classical margin bounds will fail on near-max-margin classifiers.
Our analysis provides insight on why memorization can coexist with
generalization: we show that in this challenging regime where generalization
occurs but UC fails, near-max-margin classifiers simultaneously contain some
generalizable components and some overfitting components that memorize the
data. The presence of the overfitting components is enough to preclude UC, but
the near-extremal margin guarantees that sufficient generalizable components
are present.