Published as a conference paper at ICLR 2020
A C ONSTRUCTIVE PREDICTION OF THE
GENERALIZATION ERROR ACROSS SCALES
Jonathan S. Ro
A Constructive Prediction of the Generalization Error Across Scales
The dependency of the generalization error of neural networks on model and
dataset size is of critical importance both in practice and for understanding
the theory of neural networks. Nevertheless, the functional form of this
dependency remains elusive. In this work, we present a functional form which
approximates well the generalization error in practice. Capitalizing on the
successful concept of model scaling (e.g., width, depth), we are able to
simultaneously construct such a form and specify the exact models which can
attain it across model/data scales. Our construction follows insights obtained
from observations conducted over a range of model/data scales, in various model
types and datasets, in vision and language tasks. We show that the form both
fits the observations well across scales, and provides accurate predictions
from small- to large-scale models and data.
Jonathan S. Rosenfeld Amir Rosenfeld Yonatan Belinkov Nir Shavit