Regularization in the optimization of deep neural networks is often critical to avoid undesirable over-fitting leading to better generalization of model. One of the most popular regularization algorithms is to impose @math penalty on the model parameters resulting in the decay of parameters, called weight-decay, and the decay rate is generally constant to all the model parameters in the course of optimization. In contrast to the previous approach based on the constant rate of weight-decay, we propose to consider the residual that measures dissimilarity between the current state of model and observations in the determination of the weight-decay for each parameter in an adaptive way, called adaptive weight-decay (AdaDecay) where the gradient norms are normalized within each layer and the degree of regularization for each parameter is determined in proportional to the magnitude of its gradient using the sigmoid function. We empirically demonstrate the effectiveness of AdaDecay in comparison to the state-of-the-art optimization algorithms using popular benchmark datasets: MNIST, Fashion-MNIST, and CIFAR-10 with conventional neural network models ranging from shallow to deep. The quantitative evaluation of our proposed algorithm indicates that AdaDecay improves generalization leading to better accuracy across all the datasets and models.
Weight-Decay: is an explicit way of regularization such that a regularization term is added into the energy function. Specifically @math -norm is used as the regularization term in order to penalize large weight values. Different with the other implicit methods, e.g., stochastic update and dropout, one can directly control the regularization effect by the weight-decay coefficient. The weight-decay coefficient is tuned by hand @cite_14 @cite_11 , or learned by Bayesian optimization @cite_22 @cite_0 . However, in contrast to recent development of adaptive methods of dropout @cite_4 @cite_13 @cite_35 @cite_26 @cite_47 and learning-rate @cite_31 @cite_36 @cite_9 in deep optimization, a constant weight-decay coefficient has been employed in usual. Layer-wise weight-decay has been considered in @cite_52 @cite_21 where different weight-decay coefficients are given for different layers of network model using the variance of gradients in layer. The drawback of the layer-wise method @cite_52 @cite_21 is that it assumes that layers are aligned in a single sequence. The skip-connection @cite_7 @cite_12 @cite_24 , that is one of the key architectures in the recent deep networks, makes it non-trivial.
@cite_35 @cite_31 @cite_14 @cite_4 @cite_22 @cite_26 @cite_7 @cite_36 @cite_9 @cite_21 @cite_52 @cite_0 @cite_24 @cite_47 @cite_13 @cite_12 @cite_11
2898564876 2788597178 2885062394 2583938035
Weight decay is one of the standard tricks in the neural network toolbox, but the reasons for its regularization effect are poorly understood, and recent results have cast doubt on the traditional interpretation in terms of @math regularization. Literal weight decay has been shown to outperform @math regularization for optimizers for which they differ. We empirically investigate weight decay for three optimization algorithms (SGD, Adam, and K-FAC) and a variety of network architectures. We identify three distinct mechanisms by which weight decay exerts a regularization effect, depending on the particular optimization algorithm and architecture: (1) increasing the effective learning rate, (2) approximately regularizing the input-output Jacobian norm, and (3) reducing the effective damping coefficient for second-order optimization. Our results provide insight into how to improve the regularization of neural networks. In this paper, we propose layer-wise weight decay for efficient training of deep neural networks. Our method sets different values of the weight-decay coefficients layer by layer so that the ratio of the scale of back-propagated gradients and that of the weight decay is constant throughout the network. By utilizing such a setting, we can avoid under or over-fitting and train all layers properly without having to tune the coefficients layer by layer. Experimental results show that our method can enhance the performance of existing deep neural networks without any change of network models. Multi-layer neural networks have lead to remarkable performance on many kinds of benchmark tasks in text, speech and image processing. Nonlinear parameter estimation in hierarchical models is known to be subject to overfitting and misspecification. One approach to these estimation and related problems (local minima, colinearity, feature discovery etc.) is called Dropout (Hinton, et al 2012, 2016). The Dropout algorithm removes hidden units according to a Bernoulli random variable with probability @math prior to each update, creating random "shocks" to the network that are averaged over updates. In this paper we will show that Dropout is a special case of a more general model published originally in 1990 called the Stochastic Delta Rule, or SDR (Hanson, 1990). SDR redefines each weight in the network as a random variable with mean @math and standard deviation @math . Each weight random variable is sampled on each forward activation, consequently creating an exponential number of potential networks with shared weights. Both parameters are updated according to prediction error, thus resulting in weight noise injections that reflect a local history of prediction error and local model averaging. SDR therefore implements a more sensitive local gradient-dependent simulated annealing per weight converging in the limit to a Bayes optimal network. Tests on standard benchmarks (CIFAR) using a modified version of DenseNet shows the SDR outperforms standard Dropout in test error by approx. @math with DenseNet-BC 250 on CIFAR-100 and approx. @math in smaller networks. We also show that SDR reaches the same accuracy that Dropout attains in 100 epochs in as few as 35 epochs. Deep convolutional networks have achieved successful performance in data mining field. However, training large networks still remains a challenge, as the training data may be insufficient and the model can easily get overfitted. Hence the training process is usually combined with a model regularization. Typical regularizers include weight decay, Dropout, etc. In this paper, we propose a novel regularizer, named Structured Decorrelation Constraint (SDC), which is applied to the activations of the hidden layers to prevent overfitting and achieve better generalization. SDC impels the network to learn structured representations by grouping the hidden units and encouraging the units within the same group to have strong connections during the training procedure. Meanwhile, it forces the units in different groups to learn non-redundant representations by minimizing the cross-covariance between them. Compared with Dropout, SDC reduces the co-adaptions between the hidden units in an explicit way. Besides, we propose a novel approach called Reg-Conv that can help SDC to regularize the complex convolutional layers. Experiments on extensive datasets show that SDC significantly reduces overfitting and yields very meaningful improvements on classification performance (on CIFAR-10 6.22 accuracy promotion and on CIFAR-100 9.63 promotion).