Flatten the Curve: Efficiently Training Low-Curvature Neural Networks
The highly non-linear nature of deep neural networks causes them to be
susceptible to adversarial examples and have unstable gradients which hinders
interpretability. However, existing methods to solve these issues, such as
adversarial training, are expensive and often sacrifice predictive accuracy.
  In this work, we consider curvature, which is a mathematical quantity which
encodes the degree of non-linearity. Using this, we demonstrate low-curvature
neural networks (LCNNs) that obtain drastically lower curvature than standard
models while exhibiting similar predictive performance, which leads to improved
robustness and stable gradients, with only a marginally increased training
time. To achieve this, we minimize a data-independent upper bound on the
curvature of a neural network, which decomposes overall curvature in terms of
curvatures and slopes of its constituent layers. To efficiently minimize this
bound, we introduce two novel architectural components: first, a non-linearity
called centered-softplus that is a stable variant of the softplus
non-linearity, and second, a Lipschitz-constrained batch normalization layer.
  Our experiments show that LCNNs have lower curvature, more stable gradients
and increased off-the-shelf adversarial robustness when compared to their
standard high-curvature counterparts, all without affecting predictive
performance. Our approach is easy to use and can be readily incorporated into
existing neural network models.