Modern deep artificial neural networks have achieved impressive results through models with orders of magnitude more parameters than training examples which control overfitting with the help of regularization. Regularization can be implicit, as is the case of stochastic gradient descent and parameter sharing in convolutional layers, or explicit. Explicit regularization techniques, most common forms are weight decay and dropout, have proven successful in terms of improved generalization, but they blindly reduce the effective capacity of the model, introduce sensitive hyper-parameters and require deeper and wider architectures to compensate for the reduced capacity. In contrast, data augmentation techniques exploit domain knowledge to increase the number of training examples and improve generalization without reducing the effective capacity and without introducing model-dependent parameters, since it is applied on the training data. In this paper we systematically contrast data augmentation and explicit regularization on three popular architectures and three data sets. Our results demonstrate that data augmentation alone can achieve the same performance or higher as regularized models and exhibits much higher adaptability to changes in the architecture and the amount of training data. One of the central issues in machine learning research and application is finding ways of improving generalization. Regularization, loosely defined as any modification applied to a learning algorithm that helps prevent overfitting, plays therefore a key role in machine learning (Girosi et al., 1995; Müller, 2012) . In the case of deep learning, where neural networks tend to have several orders of magnitude more parameters than training examples, statistical learning theory (Vapnik & Chervonenkis, 1971) indicates that regularization becomes even more crucial. Accordingly, a myriad of techniques have been proposed as regularizers: weight decay (Hanson & Pratt, 1989) and other L p penalties; dropout (Srivastava et al., 2014) and stochastic depth (Huang et al., 2016) , to name a few examples. Moreover, whereas in simpler machine learning algorithms the regularizers can be easily identified as explicit terms in the objective function, in modern deep neural networks the sources of regularization are not only explicit, but implicit (Neyshabur et al., 2014) . In this regard, many techniques have been studied for their regularization effect, despite not being explicitly intended as such. That is the case of unsupervised pre-training (Erhan et al., 2010) , multi-task learning (Caruana, 1998) , convolutional layers (LeCun et al., 1990) , batch normalization (Ioffe & Szegedy, 2015) or adversarial training (Szegedy et al., 2013) . In sum, there are multiple elements in deep learning that contribute to reduce overfitting and thus improve generalization. Driven by the success of such techniques and the efficient use of GPUs, considerable research effort has been devoted to finding ways of training deeper and wider networks with larger capacity (Simonyan & Zisserman, 2014; He et al., 2016; Zagoruyko & Komodakis, 2016) . Ironically, the increased representational capacity is eventually reduced in practice by the use of explicit regularization, most commonly weight decay and dropout. It is known, for instance, that the gain in generalization provided by dropout comes at the cost of using larger models and training for longer (Goodfellow et al., 2016) . Hence, it seems that with these standard regularization methods deep networks are wasting capacity (Dauphin & Bengio, 2013) . Unlike explicit regularization, data augmentation improves generalization without reducing the capacity of the model. Data augmentation, that is synthetically expanding a data set by apply-ing transformations on the available examples, has been long used in machine learning (Simard et al., 1992) and identified as a critical component of many recent successful models, like AlexNet (Krizhevsky et al., 2012) , All-CNN (Springenberg et al., 2014) or ResNet (He et al., 2016) , among others. Although it is most popular in computer vision, data augmentation has also proven effective in speech recognition (Jaitly & Hinton, 2013) , music source separation (Uhlich et al., 2017) or text categorization (Lu et al., 2006) . Today, data augmentation is an almost ubiquitous technique in deep learning, which can also be regarded as an implicit regularizer for it improves generalization. Recently, the deep learning community has become more aware of the importance of data augmentation (Hernández-García & König, 2018b) and new techniques, such as cutout (DeVries & Taylor, 2017a) or augmentation in the feature space (DeVries & Taylor, 2017b) , have been proposed. Very interestingly, a promising avenue for future research has been set by recently proposed models that automatically learn the data transformations (Hauberg et al., 2016; Lemley et al., 2017; Ratner et al., 2017; Antoniou et al., 2017) . Nonetheless, another study by Perez & Wang (2017) analyzed the performance of different techniques for object recognition and concluded that one of the most successful techniques so far is still the traditional data augmentation carried out in most studies. However, despite its popularity, the literature lacks, to our knowledge, a systematic analysis of the impact of data augmentation on convolutional neural networks compared to explicit regularization. It is a common practice to train the models with both explicit regularization, typically weight decay and dropout, and data augmentation, assuming they all complement each other. Zhang et al. (2017) included data augmentation in their analysis of generalization of deep networks, but it was questionably considered an explicit regularizer similar to weight decay and dropout. To our knowledge, the first time data augmentation and explicit regularization were systematically contrasted was the preliminary study by Hernández-García & König (2018b) . The present work aims at largely extending that work both with more empirical results and a theoretical discussion. Our specific contributions are the following: • Propose definitions of explicit and implicit regularization that aim at solving the ambiguity in the literature (Section 2). • A theoretical discussion based on statistical learning theory about the differences between explicit regularization and data augmentation, highlighting the advantages of the latter (Section 3). • An empirical analysis of the performance of models trained with and without explicit regularization, and different levels of data augmentation on several benchmarks (Sections 4 and 5). Further, we study their adaptability to learning from fewer examples (Section 5.2) and to changes in the architecture (Section 5.3). • A discussion on why encouraging data augmentation instead of explicit regularization can benefit both theory and practice in deep learning (Section 6). 2 EXPLICIT AND IMPLICIT REGULARIZATION Zhang et al. (2017) raised the thought-provoking idea that "explicit regularization may improve generalization performance, but is neither necessary nor by itself sufficient for controlling generalization error." The authors came to this conclusion from the observation that turning off the explicit regularizers of a model does not prevent the model from generalizing reasonably well. This contrasts with traditional machine learning involving convex optimization, where regularization is necessary to avoid overfitting and generalize (Vapnik & Chervonenkis, 1971) . Such observation led the authors to suggest the need for "rethinking generalization" in order to understand deep learning. We argue it is not necessary to rethink generalization if we instead rethink regularization and, in particular, data augmentation. Despite their thorough analysis and relevant conclusions, Zhang et al. (2017) arguably underestimated the role of implicit regularization and considered data augmentation an explicit form of regularization much like weight decay and dropout. This illustrates that the terms explicit and implicit regularization have been used subjectively and inconsistently in the literature before. In order to avoid the ambiguity and facilitate the discussion, we propose the following definitions of explicit and implicit regularization 1 : • Explicit regularization techniques are those which reduce the representational capacity of the model they are applied on. That is, given a model class H 0 , for instance a neural network architecture, the introduction of explicit regularization will span a new hypothesis set H 1 , which is a proper subset of the original set, i.e. H 1 H 0 . • Implicit regularization is the reduction of the generalization error or overfitting provided by means other than explicit regularization techniques. Elements that provide implicit regularization do not reduce the representational capacity, but may affect the effective capacity of the model, that is the achievable set of hypotheses given the model, the optimization algorithm, hyperparameters, etc. One of the most common explicit regularization techniques in machine learning is L p -norm regularization, of which weight decay is a particular case, widely used in deep learning. Weight decay sets a penalty on the L 2 norm of the learnable parameters, thus constraining the representational capacity of the model. Dropout is another common example of explicit regularization, where the hypothesis set is reduced by stochastically deactivating a number of neurons during training. Similar to dropout, stochastic depth, which drops whole layers instead of neurons, is also an explicit regularization technique. There are multiple elements in deep neural networks that implicitly regularize the models. Note, in this regard, that the above definition, contrary to explicit regularization, does not refer to techniques, but to a regularization effect, as it can be provided by elements of very different nature. For instance, stochastic gradient descent (SGD) is known to have an implicit regularization effect without constraining the representational capacity. Batch normalization does not either reduce the capacity, but it improves generalization by smoothing the optimization landscape Santurkar et al. (2018) . Of quite a different nature, but still implicit, is the regularization effect provided by early stopping, which does not reduce the representational, but the effective capacity. By analyzing the literature, we identified some previous pieces of work which, lacking a definition of explicit and implicit regularization, made a distinction apparently based on the mere intention of the practitioner. Under such notion, data augmentation has been considered in some cases an explicit regularization technique, as in Zhang et al. (2017) . Here, we have provided definitions for explicit and implicit regularization based on their effect on the representational capacity and argue that data augmentation is not explicit, but implicit regularization, since it does not affect the representational capacity of the model. We have presented a systematic analysis of the role of data augmentation in deep convolutional neural networks for object recognition, focusing on the comparison with popular explicit regularization techniques-weight decay and dropout. In order to facilitate the discussion and the analysis, we first proposed in Section 2 definitions of explicit and implicit regularization, which have been ambiguously used in the literature. Accordingly, we have argued that data augmentation should not be considered an explicit regularizer, such as weight decay and dropout. Then, we provided some theoretical insights in Section 3 that highlight some advantages of data augmentation over explicit regularization. Finally, we have empirically shown that explicit regularization is not only unnecessary (Zhang et al., 2017) , but also that its generalization gain can be achieved by data augmentation alone. Moreover, we have demonstrated that, unlike data augmentation, weight decay and dropout exhibit poor adaptability to changes in the architecture and the amount of training data. Despite the limitations of our empirical study, we have chosen three significantly distinct network architectures and three data sets in order to increase the generality of our conclusions, which should ideally be confirmed by future work on a wider range of models, data sets and even other domains such text or speech. It is important to note, however, that we have taken a conservative approach in our experimentation: all the hyperparameters have been kept as in the original models, which included both weight decay and dropout, as well as light augmentation. This setup is clearly suboptimal for models trained without explicit regularization. Besides, the heavier data augmentation scheme was deliberately not optimized to improve the performance and it was not the scope of this work to propose a specific data augmentation technique. As future work, we plan to propose data augmentation schemes that can more successfully be exploited by any deep model. The relevance of our findings lies in the fact that explicit regularization is currently the standard tool to enable the generalization of most machine learning methods and is included in most convolutional neural networks. However, we have empirically shown that simply removing the explicit regularizers often improves the performance or only marginally reduces it, if some data augmentation is applied. These results are supported by the theoretical insights provided in in Section 3. Zhang et al. (2017) suggested that regularization might play a different role in deep learning, not fully explained by statistical learning theory (Vapnik & Chervonenkis, 1971) . We have argued instead that the theory still naturally holds in deep learning, as long as one considers the crucial role of implicit regularization: explicit regularization seems to be no longer necessary because its contribution is already provided by the many elements that implicitly and successfully regularize the models: to name a few, stochastic gradient descent, convolutional layers and data augmentation.
Deep neural networks trained with data augmentation do not require any other explicit regularization (such as weight decay and dropout) and exhibit greater adaptaibility to changes in the architecture and the amount of training data.
{"src": "scitldr-AIC_train", "paper_id": "H1eqOnNYDH"}