Two distinct limits for deep learning as the net width @math have been proposed, depending on how the weights of the last layer scale with @math . In the "lazy-learning" regime, the dynamics becomes linear in the weights and is described by a Neural Tangent Kernel @math . By contrast, in the "feature-learning" regime, the dynamics can be expressed in terms of the density distribution of the weights. Understanding which regime describes accurately practical architectures and which one leads to better performance remains a challenge. We answer these questions and produce new characterizations of these regimes for the MNIST data set, by considering deep nets @math whose last layer of weights scales as @math at initialization, where @math is a parameter we vary. We performed systematic experiments on two setups (A) fully-connected Softplus momentum full batch and (B) convolutional ReLU momentum stochastic. We find that (1) @math separates the two regimes. (2) for (A) and (B) feature learning outperforms lazy learning, a difference in performance that decreases with @math and becomes hardly detectable asymptotically for (A) but is very significant for (B). (3) In both regimes, the fluctuations @math induced by initial conditions on the learned function follow @math , leading to a performance that increases with @math . This improvement can be instead obtained at intermediate @math values by ensemble averaging different networks. (4) In the feature regime there exists a time scale @math , such that for @math the dynamics is linear. At @math , the output has grown by a magnitude @math and the changes of the tangent kernel @math become significant. Ultimately, it follows @math for ReLU and Softplus activation, with @math & @math when depth grows.
Deep neural networks are successful at a variety of task, yet understanding why they work remains a challenge. A surprising observation is that their performance on supervised tasks keeps increasing with their width @math in the over-parametrized regime where they already fit all the training data @cite_18 @cite_22 @cite_20 @cite_1 . This fact underlines the importance of describing deep learning in the limit @math , which has received considerable interest recently. When all the weights @math of the network are initialized with a value of order @math at initialization, the transmission of the signal through the network in the infinite-width limit is now well understood. The output function @math is a Gaussian random processes with some covariance that can be computed @cite_7 @cite_27 @cite_17 @cite_3 @cite_10 @cite_19 .
@cite_18 @cite_22 @cite_7 @cite_1 @cite_3 @cite_19 @cite_27 @cite_10 @cite_20 @cite_17
2798986185 2214429195 2907047316 2899748887
Neural networks, a central tool in machine learning, have demonstrated remarkable, high fidelity performance on image recognition and classification tasks. These successes evince an ability to accurately represent high dimensional functions, potentially of great use in computational and applied mathematics. That said, there are few rigorous results about the representation error and trainability of neural networks. Here we characterize both the error and the scaling of the error with the size of the network by reinterpreting the standard optimization algorithm used in machine learning applications, stochastic gradient descent, as the evolution of a particle system with interactions governed by a potential related to the objective or "loss" function used to train the network. We show that, when the number @math of parameters is large, the empirical distribution of the particles descends on a convex landscape towards a minimizer at a rate independent of @math . We establish a Law of Large Numbers and a Central Limit Theorem for the empirical distribution, which together show that the approximation error of the network universally scales as @math . Remarkably, these properties do not depend on the dimensionality of the domain of the function that we seek to represent. Our analysis also quantifies the scale and nature of the noise introduced by stochastic gradient descent and provides guidelines for the step size and batch size to use when training a neural network. We illustrate our findings on examples in which we train neural network to learn the energy function of the continuous 3-spin model on the sphere. The approximation error scales as our analysis predicts in as high a dimension as @math . Deep neural networks have achieved impressive supervised classification performance in many tasks including image recognition, speech recognition, and sequence to sequence learning. However, this success has not been translated to applications like question answering that may involve complex arithmetic and logic reasoning. A major limitation of these models is in their inability to learn even simple arithmetic and logic operations. For example, it has been shown that neural networks fail to learn to add two binary numbers reliably. In this work, we propose Neural Programmer, an end-to-end differentiable neural network augmented with a small set of basic arithmetic and logic operations. Neural Programmer can call these augmented operations over several steps, thereby inducing compositional programs that are more complex than the built-in operations. The model learns from a weak supervision signal which is the result of execution of the correct program, hence it does not require expensive annotation of the correct program itself. The decisions of what operations to call, and what data segments to apply to are inferred by Neural Programmer. Such decisions, during training, are done in a differentiable fashion so that the entire network can be trained jointly by gradient descent. We find that training the model is difficult, but it can be greatly improved by adding random noise to the gradient. On a fairly complex synthetic table-comprehension dataset, traditional recurrent networks and attentional models perform poorly while Neural Programmer typically obtains nearly perfect accuracy. The advent of deep learning is a breakthrough in artificial intelligence, for which a theoretical understanding is lacking. Supervised deep learning involves the training of neural networks with a large number @math of parameters. For large enough @math , in the so-called over-parametrized regime, one can essentially fit the training data points. Sparsity-based arguments would suggest that the generalization error increases as @math grows past a certain threshold @math . Instead, empirical studies have shown that in the over-parametrized regime, generalization error keeps decreasing with @math . We resolve this paradox, through a new framework. We rely on the so-called Neural Tangent Kernel, which connects large neural nets to kernel methods, to show that the initialization causes finite-size random fluctuations @math of the neural net output function @math around its expectation @math . These affect the generalization error @math for classification: under natural assumptions, it decays to a plateau value @math in a power-law fashion @math . This description breaks down at a so-called jamming transition @math . At this threshold, we argue that @math diverges. This result leads to a plausible explanation for the cusp in test error known to occur at @math . Our results are confirmed by extensive empirical observations on the MNIST and CIFAR image datasets. Our analysis finally suggests that, given a computational envelope, it is best to use several nets of intermediate sizes, just beyond @math , and to average their outputs. Deep neural networks (DNNs) have demonstrated dominating performance in many fields; since AlexNet, networks used in practice are going wider and deeper. On the theoretical side, a long line of works has been focusing on training neural networks with one hidden layer. The theory of multi-layer networks remains largely unsettled. In this work, we prove why stochastic gradient descent (SGD) can find @math on the training objective of DNNs in @math . We only make two assumptions: the inputs are non-degenerate and the network is over-parameterized. The latter means the network width is sufficiently large: @math in @math , the number of layers and in @math , the number of samples. Our key technique is to derive that, in a sufficiently large neighborhood of the random initialization, the optimization landscape is almost-convex and semi-smooth even with ReLU activations. This implies an equivalence between over-parameterized neural networks and neural tangent kernel (NTK) in the finite (and polynomial) width setting. As concrete examples, starting from randomly initialized weights, we prove that SGD can attain 100 training accuracy in classification tasks, or minimize regression loss in linear convergence speed, with running time polynomial in @math . Our theory applies to the widely-used but non-smooth ReLU activation, and to any smooth and possibly non-convex loss functions. In terms of network architectures, our theory at least applies to fully-connected neural networks, convolutional neural networks (CNN), and residual neural networks (ResNet).