Intrinsic dimensionality and generalization properties of the $\mathcal{R}$-norm inductive bias
We study the structural and statistical properties of $\mathcal{R}$-norm
minimizing interpolants of datasets labeled by specific target functions. The
$\mathcal{R}$-norm is the basis of an inductive bias for two-layer neural
networks, recently introduced to capture the functional effect of controlling
the size of network weights, independently of the network width. We find that
these interpolants are intrinsically multivariate functions, even when there
are ridge functions that fit the data, and also that the $\mathcal{R}$-norm
inductive bias is not sufficient for achieving statistically optimal
generalization for certain learning problems. Altogether, these results shed
new light on an inductive bias that is connected to practical neural network
training.