Adversarial examples are a challenging open problem for deep neural networks. We propose in this paper to add a penalization term that forces the decision function to be at in some regions of the input space, such that it becomes, at least locally, less sensitive to attacks. Our proposition is theoretically motivated and shows on a rst set of carefully conducted experiments that it behaves as expected when used alone, and seems promising when coupled with adversarial training.
The difference between double backpropagation and our gradient penalization is that we penalize the @math -norm of the gradient of each output with respect to the input while, in backpropagation, the @math -norm of the loss is penalized. Hence, we should not have the problem that occurs when the penalization term is multiplied by a small error vector. Nevertheless, this might not be a problem when using a different loss function. Although empirical evidence show double backpropagation's efficiency to enhance generalization, it is insufficient to defend against adversarial examples. That is what @cite_1 highlights saying limiting sensitivity to infinitesimal perturbation [e.g., using double backpropagation][] drucker1992improving only provides constraints very near training examples, so it does not solve the adversarial perturbation problem . But there are evidence that coupling that gradient penalty with adversarial training increases the robustness.
@cite_1
2896834587
We study the implicit bias of gradient descent methods in solving a binary classification problem over a linearly separable dataset. The classifier is described by a nonlinear ReLU model and the objective function adopts the exponential loss function. We first characterize the landscape of the loss function and show that there can exist spurious asymptotic local minima besides asymptotic global minima. We then show that gradient descent (GD) can converge to either a global or a local max-margin direction, or may diverge from the desired max-margin direction in a general context. For stochastic gradient descent (SGD), we show that it converges in expectation to either the global or the local max-margin direction if SGD converges. We further explore the implicit bias of these algorithms in learning a multi-neuron network under certain stationary conditions, and show that the learned classifier maximizes the margins of each sample pattern partition under the ReLU activation.