Dropout is a popular regularization technique in neural networks. Yet, the reason for its success is still not fully understood. This paper provides a new interpretation of Dropout from a frame theory perspective. By drawing a connection to recent developments in analog channel coding, we suggest that for a certain family of autoencoders with a linear encoder, the minimizer of an optimization with dropout regularization on the encoder is an equiangular tight frame (ETF). Since this optimization is non-convex, we add another regularization that promotes such structures by minimizing the cross-correlation between filters in the network. We demonstrate its applicability in convolutional and fully connected layers in both feed-forward and recurrent networks. All these results suggest that there is indeed a relationship between dropout and ETF structure of the regularized linear operations.
Weight decay @cite_2 , is a regularization that minimizes the @math norm of the layer weights, intentionally reducing their flexibility. Replacing the @math norm with @math has been also explored @cite_59 @cite_34 . While the latter induces sparse weight matrices, it is harder to optimize, yielding weight decay a more commonly used method. Another alternative to weight decay is Jacobian regularization. It has been proposed for autoencoders in @cite_31 , and in @cite_33 suggested for general neural networks showing that it increases the input margin in them, and thus improving their performance.
@cite_33 @cite_59 @cite_2 @cite_31 @cite_34
2898564876 2144513243 2125059447 2788597178
Weight decay is one of the standard tricks in the neural network toolbox, but the reasons for its regularization effect are poorly understood, and recent results have cast doubt on the traditional interpretation in terms of @math regularization. Literal weight decay has been shown to outperform @math regularization for optimizers for which they differ. We empirically investigate weight decay for three optimization algorithms (SGD, Adam, and K-FAC) and a variety of network architectures. We identify three distinct mechanisms by which weight decay exerts a regularization effect, depending on the particular optimization algorithm and architecture: (1) increasing the effective learning rate, (2) approximately regularizing the input-output Jacobian norm, and (3) reducing the effective damping coefficient for second-order optimization. Our results provide insight into how to improve the regularization of neural networks. It has been observed in numerical simulations that a weight decay can improve generalization in a feed-forward neural network. This paper explains why. It is proven that a weight decay has two effects in a linear network. First, it suppresses any irrelevant components of the weight vector by choosing the smallest vector that solves the learning problem. Second, if the size is chosen right, a weight decay can suppress some of the effects of static noise on the targets, which improves generalization quite a lot. It is then shown how to extend these results to networks with hidden layers and non-linear units. Finally the theory is confirmed by some numerical simulations using the data from NetTalk. Weight decay was proposed to reduce overfitting which often appears in the generalization tasks of artificial neural nets. Here weight decay is applied to a well defined model system based on a single layer perceptron, which exhibits strong overfitting. Since we know for this system the optimal nonoverfitting solution we can compare the effect of the weight decay with this solution. A strategy to find the optimal weight decay strength, which leads to the optimal solution for any number of examples, is proposed. In this paper, we propose layer-wise weight decay for efficient training of deep neural networks. Our method sets different values of the weight-decay coefficients layer by layer so that the ratio of the scale of back-propagated gradients and that of the weight decay is constant throughout the network. By utilizing such a setting, we can avoid under or over-fitting and train all layers properly without having to tune the coefficients layer by layer. Experimental results show that our method can enhance the performance of existing deep neural networks without any change of network models.