[Shrinked]
Our regulariser relies on a measure of the model's sensitivity to relative changes in the values of similar features. One way we
quantify this sensitivity is through the model's Jacobian with respect to the input features. Regularisers on the Jacobian
are used when one wants to stabilise/robustify models to noisy inputs. Relevant work includes
contractive auto encoders, [1]}, manifold tangent classifiers, , and [2]}.
Such regularisers are typically matrix norms, e.g. Frobenius, of the Jacobian at the input instances. Their effect is to
force the model to be relatively
constant in small neighbors around the input instances, thus introducing invariance for small input variations.
[1]}, [1]}, distinguish between analytical and stochastic methods to model
stabilisation, where the former rely on analytical approaches such as the Jacobian regularisation, and the latter
on explicit input corruption and data augmentation. Denoising autoencoders, [5]} follow the stochastic paradigm and
require that small random variations in the inputs have only a limited effect on the model output.
Optimizing the Jacobian in networks with more than one layer is cumbersome, thus very often the stochastic
approach is prefered over the analytic e.g. [2]}. Regularisers on higher order derivatives, Hessian,
are also used, [7]}, in such cases the stochastic approach is the only choice due to
the prohibitive cost of optimizing the Hessian term.
[8]}, [8]}, also used Gaussian perturbations
to stabilise the network's output with respect to variations in the input,
essentially augmenting the training data. The authors note that unlike typical data augmentation approaches they do not evaluate
the augmented instances with the loss function but only with the stability regulariser because they have observed that the opposite lead
to underfitting.

[Alexandros]Need to discuss the data augmentation for stability to noise in to more detail.
The manifold tangent classifiers, , uses the Jacobian of the last layer but it seems to be much
more elaborate than the other methods. Definetely to look at more closely.
