Despite the impressive performance of deep neural networks (DNNs) on numerous learning  tasks, they still exhibit uncouth behaviours. One puzzling behaviour is the subtle sensitive reaction of DNNs to various noise attacks. Such a nuisance has strengthened the line of research around developing and training noise-robust networks. In this work, we propose a new training regularizer that aims to minimize the probabilistic expected training loss of a DNN subject to a generic Gaussian input. We provide an efficient and simple approach to approximate such a regularizer for arbitrarily deep networks. This is done by leveraging the analytic expression of the output mean of a shallow neural network, avoiding the need for memory and computation expensive data augmentation. We conduct extensive experiments on LeNet and AlexNet on various datasets including MNIST, CIFAR10, and CIFAR100 to demonstrate the effectiveness of our proposed regularizer. In particular, we show that networks that are trained with the proposed regularizer benefit from a boost in robustness against Gaussian noise to an equivalent amount of performing 3-21 folds of noisy data augmentation. Moreover, we empirically show on several architectures and datasets that improving robustness against Gaussian noise, by using the new regularizer, can improve the overall robustness against 6 other types of attacks by two orders of magnitude. Deep neural networks (DNNs) have emerged as generic models that can be trained to perform impressively well in a variety of learning tasks ranging from object recognition (He et al., 2016) and semantic segmentation (Long et al., 2015) to speech recognition and bioinformatics (Angermueller et al., 2016) . Despite their increasing popularity, flexibility, generality, and performance, DNNs have been recently shown to be quite susceptible to small imperceptible input noise (Szegedy et al., 2014; Moosavi-Dezfooli et al., 2016; Goodfellow et al., 2015) . Such analysis gives a clear indication that even state-of-the-art DNNs may lack robustness. Consequently, there has been an ever-growing interest in the machine learning community to study this uncanny behaviour. In particular, the work of (Goodfellow et al., 2015; Moosavi-Dezfooli et al., 2016) demonstrates that there are systematic approaches to constructing adversarial attacks that result in misclassification errors with high probability. Even more peculiarly, some noise perturbations seem to be doubly agnostic (Moosavi-Dezfooli et al., 2017) , i.e. there exist deterministic perturbations that can result in misclassification errors with high probability when applied to different networks, irrespective of the input (denoted network and input agnostic). Understanding this degradation in performance under adversarial attacks is of tremendous importance, especially for real-world DNN deployment, e.g. self-driving cars/drones and equipment for the visually impaired. A standard and popular means to alleviate this nuisance is noisy data augmentation in training, i.e. a DNN is exposed to noisy input images during training so as to bolster its robustness during inference. Several works have demonstrated that DNNs can in fact benefit from such augmentation (Moosavi-Dezfooli et al., 2016; Goodfellow et al., 2015) . However, data augmentation in general might not be sufficient for two reasons. (1) Particularly with high-dimensional input noise, the amount of data augmentation necessary to sufficiently capture the noise space will be very large, which will increase training time. (2) Data augmentation with high energy noise can negatively impact the performance on noise-free test examples. This can be explained by the fundamental trade-off between accuracy and robustness (Tsipras et al., 2018; Boopathy et al., 2019) . It can also arise from the fact that augmentation forces the DNN to have the same prediction for two vastly different versions of the same input, noise-free and a substantially corrupted version. Addressing the sensitivity problem of deep neural networks to adversarial perturbation is of great importance to the machine learning community. However, building robust classifiers against this noises is computationally expensive, as it is generally done through the means of data augmentation. We propose a generic lightweight analytic regularizer, which can be applied to any deep neural network with a ReLU activation after the first affine layer. It is designed to increase the robustness of the trained models under additive Gaussian noise. We demonstrate this with multiple architectures and datasets and show that it outperforms data augmentation without observing any noisy examples. A EXPERIMENTAL SETUP AND DETAILS. All experiments, are conducted using PyTorch version 0.4.1 Paszke et al. (2017) . All hyperparameters are fixed and Table 2 we report the setup for the two optimizers. In particular, we use the Adam optimizaer Kingma & Ba (2015) with β 1 = 0.9, β 2 = 0.999, = 10 −8 with amsgrad set to False. The second optimizer is SGD Loshchilov & Hutter (2017) with momentum=0.9, dampening=0, with Nesterov acceleration. In each experiment, we randomly split the training dataset into 10% validation and 90% training and monitor the validation loss after each epoch. If validation loss did not improve for lr patience epochs, we reduce the learning rate by multiplying it by lr factor. We start with an initial learning rate of lr initial. The training is terminated only if the validation loss did not improve for loss patience number of epochs or if the training reached 100 epochs. We report the results of the model with the best validation loss. In particular, one can observe that with σ large than 0.7 the among of noise is severe even for the human level. Training on such extreme noise levels will deem data augmentation to be difficult.
An efficient estimate to the Gaussian first moment of DNNs as a regularizer to training robust networks.
{"src": "scitldr-AIC_train", "paper_id": "B1xDq2EFDH"}