A Quadrature Perspective on Frequency Bias in Neural Network Training with Nonuniform Data
Small generalization errors of over-parameterized neural networks (NNs) can
be partially explained by the frequency biasing phenomenon, where
gradient-based algorithms minimize the low-frequency misfit before reducing the
high-frequency residuals. Using the Neural Tangent Kernel (NTK), one can
provide a theoretically rigorous analysis for training where data are drawn
from constant or piecewise-constant probability densities. Since most training
data sets are not drawn from such distributions, we use the NTK model and a
data-dependent quadrature rule to theoretically quantify the frequency biasing
of NN training given fully nonuniform data. By replacing the loss function with
a carefully selected Sobolev norm, we can further amplify, dampen,
counterbalance, or reverse the intrinsic frequency biasing in NN training.