Batch normalization (batch norm) is often used in an attempt to stabilize and accelerate training in deep neural networks. In many cases it indeed decreases the number of parameter updates required to achieve low training error. However, it also reduces robustness to small adversarial input perturbations and noise by double-digit percentages, as we show on five standard datasets. Furthermore, substituting weight decay for batch norm is sufficient to nullify the relationship between adversarial vulnerability and the input dimension. Our work is consistent with a mean-field analysis that found that batch norm causes exploding gradients. Batch norm is a standard component of modern deep neural networks, and tends to make the training process less sensitive to the choice of hyperparameters in many cases (Ioffe & Szegedy, 2015) . While ease of training is desirable for model developers, an important concern among stakeholders is that of model robustness to plausible, previously unseen inputs during deployment.The adversarial examples phenomenon has exposed unstable predictions across state-of-the-art models (Szegedy et al., 2014) . This has led to a variety of methods that aim to improve robustness, but doing so effectively remains a challenge BID0 Schott et al., 2019; Hendrycks & Dietterich, 2019; Jacobsen et al., 2019) . We believe that a prerequisite to developing methods that increase robustness is an understanding of factors that reduce it.Presented at the ICML 2019 Workshop on Identifying and Understanding Deep Learning Phenomena. Copyright 2019 by the author(s). We found that there is no free lunch with batch norm: the accelerated training properties and occasionally higher clean test accuracy come at the cost of robustness, both to additive noise and for adversarial perturbations. We have shown that there is no inherent relationship between the input dimension and vulnerability. Our results highlight the importance of identifying the disparate mechanisms of regularization techniques, especially when concerned about robustness.Bjorck, N., Gomes, C. P., Selman, B., and Weinberger, K. Q.Understanding
Batch normalization reduces adversarial robustness, as well as general robustness in many cases, particularly to noise corruptions.
{"src": "scitldr-AIC_train", "paper_id": "BkxOwVShhE"}