We propose a framework to understand the unprecedented performance and robustness of deep neural networks using field theory. Correlations between the weights within the same layer can be described by symmetries in that layer, and networks generalize better if such symmetries are broken to reduce the redundancies of the weights. Using a two parameter field theory, we find that the network can break such symmetries itself towards the end of training in a process commonly known in physics as spontaneous symmetry breaking. This corresponds to a network generalizing itself without any user input layers to break the symmetry, but by communication with adjacent layers. In the layer decoupling limit applicable to residual networks (He et al., 2015), we show that the remnant symmetries that survive the non-linear layers are spontaneously broken based on empirical results. The Lagrangian for the non-linear and weight layers together has striking similarities with the one in quantum field theory of a scalar. Using results from quantum field theory we show that our framework is able to explain many experimentally observed phenomena, such as training on random labels with zero error (Zhang et al., 2017), the information bottleneck and the phase transition out of it (Shwartz-Ziv & Tishby, 2017), shattered gradients (Balduzzi et al., 2017), and many more. Deep neural networks have been used in image recognition tasks with great success. The first of its kind, AlexNet BID8 , led to many other neural architectures have been proposed to achieve start-of-the-art results in image processing at the time. Some of the notable architectures include, VGG BID12 , Inception BID14 and Residual networks (ResNet) BID3 .Understanding the inner workings of deep neural networks remains a difficult task. It has been discovered that the training process ceases when it goes through an information bottleneck (ShwartzZiv & Tishby, 2017 ) until learning rate is decreased to a suitable amount then the network under goes a phase transition. Deep networks appear to be able to regularize themselves and able to train on randomly labeled data BID18 with zero training error. The gradients in deep neural networks behaves as white noise over the layers BID1 . And many other unexplained phenomena .A recent work BID0 showed that the ensemble behavior and binomial path lengths BID15 of ResNets can be explained by just a Taylor series expansion to first order in the decoupling limit. They found that the series approximation generates a symmetry breaking layer that reduces the redundancy of weights, leading to a better generalization. Because the ResNet does not contain such symmetry breaking layers in the architecture. They suggest that ResNets are able to break the symmetry by the communication between the layers. Another recent work also employed the Taylor expansion to investigate ResNets BID6 .In statistical terms, a quantum theory describes errors from the mean of random variables. We wish to study how error propagate through each layer in the network, layer by layer. In the limit of a continuous sample space, the quantum theory becomes a quantum field theory. The effects of sampling error and labelling error can then be investigated. It is well known in physics that a scalar field can drive a phase transition. Using a scalar field theory we show that a phase transition must exist towards the end of training based on empirical results. It is also responsible for the remarkable performance of deep networks compared to other classical models. In Appendix D, We explain that quantum field theory is likely one of the simplest model that can describe a deep network layer by layer in the decoupling limit.Much of the literature on neural network design focuses on different neural architecture that breaks symmetry explicitly, rather than spontaneously. For instance, non-linear layers explicitly breaks the symmetry of affine transformations. There is little discussion on spontaneous symmetry breaking. In neural networks, the Goldstone theorem in field theory states that for every continuous symmetry that is spontaneously broken, there exists a weight with zero Hessian eigenvalue at the loss minimum. No such weights would appear if the symmetries are explicitly broken . It turns out that many seemingly different experimental results can be explained by the presence of these zero eigenvalue weights. In this work, we exploit the layer decoupling limit applicable to ResNets to approximate the loss functions with a power series in symmetry invariant quantities and illustrate that spontaneous symmetry breaking of affine symmetries is the sufficient and necessary condition for a deep network to attain its unprecedented power.The organization of this paper is as follows. The background on deep neural networks and field theory is given in Section 2. Section 3 shows that remnant symmetries can exist in a neural network and that the weights can be approximated by a scalar field. Experimental results that confirm our theory is given in Section 4. We summarize more evidence from other experiments in Appendix A. A review of field theory is given in Appendix B. An explicit example of spontaneous symmetry breaking is shown in Appendix C.
Closed form results for deep learning in the layer decoupling limit applicable to Residual Networks
{"src": "scitldr-AIC_train", "paper_id": "SkiCjzNTZ"}