Critical Investigation of Failure Modes in Physics-informed Neural Networks
Several recent works in scientific machine learning have revived interest in
the application of neural networks to partial differential equations (PDEs). A
popular approach is to aggregate the residual form of the governing PDE and its
boundary conditions as soft penalties into a composite objective/loss function
for training neural networks, which is commonly referred to as physics-informed
neural networks (PINNs). In the present study, we visualize the loss landscapes
and distributions of learned parameters and explain the ways this particular
formulation of the objective function may hinder or even prevent convergence
when dealing with challenging target solutions. We construct a purely
data-driven loss function composed of both the boundary loss and the domain
loss. Using this data-driven loss function and, separately, a physics-informed
loss function, we then train two neural network models with the same
architecture. We show that incomparable scales between boundary and domain loss
terms are the culprit behind the poor performance. Additionally, we assess the
performance of both approaches on two elliptic problems with increasingly
complex target solutions. Based on our analysis of their loss landscapes and
learned parameter distributions, we observe that a physics-informed neural
network with a composite objective function formulation produces highly
non-convex loss surfaces that are difficult to optimize and are more prone to
the problem of vanishing gradients.