Modern neural networks are over-parametrized. In particular, each rectified linear hidden unit can be modified by a multiplicative factor by adjusting input and out- put weights, without changing the rest of the network. Inspired by the Sinkhorn-Knopp algorithm, we introduce a fast iterative method for minimizing the l2 norm of the weights, equivalently the weight decay regularizer. It provably converges to a unique solution. Interleaving our algorithm with SGD during training improves the test accuracy. For small batches, our approach offers an alternative to batch- and group- normalization on CIFAR-10 and ImageNet with a ResNet-18. Deep Neural Networks (DNNs) have achieved outstanding performance across a wide range of empirical tasks such as image classification BID1 , image segmentation (He et al., 2017) , speech recognition (Hinton et al., 2012a) , natural language processing (Collobert et al., 2011) or playing the game of Go BID16 . These successes have been driven by the availability of large labeled datasets such as ImageNet BID13 , increasing computational power and the use of deeper models (He et al., 2015b) .Although the expressivity of the function computed by a neural network grows exponentially with depth BID12 Telgarsky, 2016) , in practice deep networks are vulnerable to both over-and underfitting (Glorot & Bengio, 2010; BID1 He et al., 2015b) . Widely used techniques to prevent DNNs from overfitting include regularization methods such as weight decay BID2 , Dropout (Hinton et al., 2012b) and various data augmentation schemes BID1 BID17 BID19 He et al., 2015b) . Underfitting can occur if the network gets stuck in a local minima, which can be avoided by using stochastic gradient descent algorithms (Bottou, 2010; Duchi et al., 2011; BID18 BID0 , sometimes along with carefully tuned learning rate schedules (He et al., 2015b; Goyal et al., 2017) .Training deep networks is particularly challenging due to the vanishing/exploding gradient problem. It has been studied for Recurrent Neural networks (RNNs) (Hochreiter et al., 2001 ) as well as standard feedforward networks (He et al., 2015a; BID7 . After a few iterations , the gradients computed during backpropagation become either too small or too large, preventing the optimization scheme from converging. This is alleviated by using non-saturating activation functions such as rectified linear units (ReLUs) BID1 or better initialization schemes preserving the variance of the input across layers (Glorot & Bengio, 2010; BID7 He et al., 2015a) . Failure modes that prevent the training from starting have been theoretically studied by Hanin & Rolnick (2018) .Two techniques in particular have allowed vision models to achieve "super-human" accuracy. Batch Normalization (BN) was developed to train Inception networks (Ioffe & Szegedy, 2015) . It introduces intermediate layers that normalize the features by the mean and variance computed within the current batch. BN is effective in reducing training time, provides better generalization capabilities after training and diminishes the need for a careful initialization. Network architectures such as ResNet (He et al., 2015b) and DenseNet (Huang et al., 2016) use skip connections along with BN to improve the information flow during both the forward and backward passes. DISPLAYFORM0 Figure 1: Matrices W k and W k+1 are updated by multiplying the columns of the first matrix with rescaling coefficients. The rows of the second matrix are inversely rescaled to ensure that the product of the two matrices is unchanged. The rescaling coefficients are strictly positive to ensure functional equivalence when the matrices are interleaved with ReLUs. This rescaling is applied iteratively to each pair of adjacent matrices. In this paper, we address the more complex cases of biases, convolutions, max-pooling or skip-connections to be able to balance modern CNN architectures.However, BN has some limitations. In particular, BN only works well with sufficiently large batch sizes (Ioffe & Szegedy, 2015; Wu & He, 2018) . For sizes below 16 or 32, the batch statistics have a high variance and the test error increases significantly. This prevents the investigation of highercapacity models because large, memory-consuming batches are needed in order for BN to work in its optimal range. In many use cases, including video recognition (Carreira & Zisserman, 2017) and image segmentation (He et al., 2017) , the batch size restriction is even more challenging because the size of the models allows for only a few samples per batch. Another restriction of BN is that it is computationally intensive, typically consuming 20% to 30% of the training time. Variants such as Group Normalization (GN) (Wu & He, 2018 ) cover some of the failure modes of BN.In this paper, we introduce a novel algorithm to improve both the training speed and generalization accuracy of networks by using their over-parameterization to regularize them. In particular, we focus on neural networks that are positive-rescaling equivalent BID8 , i.e. whose weights are identical up to positive scalings and matching inverse scalings. The main principle of our method, referred to as Equi-normalization (ENorm ), is illustrated in Figure 1 for the fullyconnected case. We scale two consecutive matrices with rescaling coefficients that minimize the joint p norm of those two matrices. This amounts to re-parameterizing the network under the constraint of implementing the same function. We conjecture that this particular choice of rescaling coefficients ensures a smooth propagation of the gradients during training.A limitation is that our current proposal, in its current form, can only handle learned skipconnections like those proposed in type-C ResNet. For this reason, we focus on architectures, in particular ResNet18, for which the learning converges with learned skip-connection, as opposed to architectures like ResNet-50 for which identity skip-connections are required for convergence.In summary,• We introduce an iterative, batch-independent algorithm that re-parametrizes the network within the space of rescaling equivalent networks, thus preserving the function implemented by the network; • We prove that the proposed Equi-normalization algorithm converges to a unique canonical parameterization of the network that minimizes the global p norm of the weights, or equivalently, when p = 2, the weight decay regularizer; • We extend ENorm to modern convolutional architectures, including the widely used ResNets, and show that the theoretical computational overhead is lower compared to BN (×50) and even compared to GN (×3); • We show that applying one ENorm step after each SGD step outperforms both BN and GN on the CIFAR-10 (fully connected) and ImageNet (ResNet-18) datasets.• Our code is available at https://github.com/facebookresearch/enorm. The paper is organized as follows. Section 2 reviews related work. Section 3 defines our Equinormalization algorithm for fully-connected networks and proves the convergence. Section 4 shows how to adapt ENorm to convolutional neural networks (CNNs). Section 5 details how to employ ENorm for training neural networks and Section 6 presents our experimental results .
Fast iterative algorithm to balance the energy of a network while staying in the same functional equivalence class
{"src": "scitldr-AIC_train", "paper_id": "r1gEqiC9FX"}