Momentum Diminishes the Effect of Spectral Bias in Physics-Informed Neural Networks
Physics-informed neural network (PINN) algorithms have shown promising
results in solving a wide range of problems involving partial differential
equations (PDEs). However, they often fail to converge to desirable solutions
when the target function contains high-frequency features, due to a phenomenon
known as spectral bias. In the present work, we exploit neural tangent kernels
(NTKs) to investigate the training dynamics of PINNs evolving under stochastic
gradient descent with momentum (SGDM). This demonstrates SGDM significantly
reduces the effect of spectral bias. We have also examined why training a model
via the Adam optimizer can accelerate the convergence while reducing the
spectral bias. Moreover, our numerical experiments have confirmed that
wide-enough networks using SGDM still converge to desirable solutions, even in
the presence of high-frequency features. In fact, we show that the width of a
network plays a critical role in convergence.