Standalone Neural ODEs with Sensitivity Analysis
This paper presents the Standalone Neural ODE (sNODE), a continuous-depth
neural ODE model capable of describing a full deep neural network. This uses a
novel nonlinear conjugate gradient (NCG) descent optimization scheme for
training, where the Sobolev gradient can be incorporated to improve smoothness
of model weights. We also present a general formulation of the neural
sensitivity problem and show how it is used in the NCG training. The
sensitivity analysis provides a reliable measure of uncertainty propagation
throughout a network, and can be used to study model robustness and to generate
adversarial attacks. Our evaluations demonstrate that our novel formulations
lead to increased robustness and performance as compared to ResNet models, and
that it opens up for new opportunities for designing and developing machine
learning with improved explainability.