Training recurrent neural networks (RNNs) that possess long-term memory is challenging. We provide insight into the trade-off between the smoothness of the cost function and the memory retention capabilities of the network. We express both aspects in terms of the Lipschitz constant of the dynamics modeled by the network. This allows us to make a distinction between three types of regions in the parameter space. In the first region, the network experiences problems in retaining long-term information, while at the same time the cost function is smooth and easy for gradient descent to navigate in. In the second region, the amount of stored information increases with time and the cost function is intricate and full of local minima. The third region is in between the two other regions and here the RNN is able to retain long-term information. Based on these theoretical findings we present the hypothesis that good parameter choices for the RNN are located in between the well-behaved and the ill-behaved cost function regions. The concepts presented in the paper are illustrated by artificially generated and real examples.
conditions for exploding gradients to appear are also provided in @cite_1 . Their result can be understood as a special case of Theorem and is limited to a single layer of vanilla RNN. By constraining the results to this vanilla case, they do not cover interesting nonlinear behaviors, such as periodic behavior and strange attractors, that are studied here. Our results are also more general, because we include bounds on second order derivatives.
@cite_1
2783402179 2591954064 1884565747 2808460953
We give a rigorous analysis of the statistical behavior of gradients in randomly initialized feed-forward networks with ReLU activations. Our results show that a fully connected depth @math ReLU net with hidden layer widths @math will have exploding and vanishing gradients if and only if @math is large. The point of view of this article is that whether a given neural net will have exploding vanishing gradients is a function mainly of the architecture of the net, and hence can be tested at initialization. Our results imply that a fully connected network that produces manageable gradients at initialization must have many hidden layers that are about as wide as the network is deep. This work is related to the mean field theory approach to random neural nets. From this point of view, we give a rigorous computation of the @math corrections to the propagation of gradients at the so-called edge of chaos. A long-standing obstacle to progress in deep learning is the problem of vanishing and exploding gradients. Although, the problem has largely been overcome via carefully constructed initializations and batch normalization, architectures incorporating skip-connections such as highway and resnets perform much better than standard feedforward architectures despite well-chosen initialization and batch normalization. In this paper, we identify the shattered gradients problem. Specifically, we show that the correlation between gradients in standard feedforward networks decays exponentially with depth resulting in gradients that resemble white noise whereas, in contrast, the gradients in architectures with skip-connections are far more resistant to shattering, decaying sublinearly. Detailed empirical evidence is presented in support of the analysis, on both fully-connected networks and convnets. Finally, we present a new "looks linear" (LL) initialization that prevents shattering, with preliminary experiments showing the new initialization allows to train very deep networks without the addition of skip-connections. We consider models given by Hamiltonians of the form View the MathML sourceH(I,f,p,q,t;e)=h(I)+?j=1n±(12pj2+Vj(qj))+eQ(I,f,p,q,t;e) Turn MathJax on where I?I?Rd,f?TdI?I?Rd,f?Td, p,q?Rnp,q?Rn, t?T1t?T1. These are higher dimensional analogues, both in the center and hyperbolic directions, of the models studied in , and and are usually called “a-priori unstable Hamiltonian systems”. All these models present the large gap problem. We show that, for 0<e«10<e«1, under regularity and explicit non-degeneracy conditions on the model, there are orbits whose action variables I perform rather arbitrary excursions in a domain of size O(1)O(1). This domain includes resonance lines and, hence, large gaps among d-dimensional KAM tori. This phenomenon is known as Arnold diffusion. The method of proof follows closely the strategy of and . The main new phenomenon that appears when the dimension d of the center directions is larger than one is the existence of multiple resonances in the space of actions I?I?RdI?I?Rd. We show that, since these multiple resonances happen in sets of codimension greater than one in the space of actions I, they can be contoured. This corresponds to the mechanism called diffusion across resonances in the Physics literature. The present paper, however, differs substantially from and . On the technical details of the proofs, we have taken advantage of the theory of the scattering map developed in [31]—notably the symplectic properties—which were not available when the above papers were written. We have analyzed the conditions imposed on the resonances in more detail. More precisely, we have found that there is a simple condition on the Melnikov potential which allows us to conclude that the resonances are crossed. In particular, this condition does not depend on the resonances. So that the results are new even when applied to the models in and In recent years, state-of-the-art methods in computer vision have utilized increasingly deep convolutional neural network architectures (CNNs), with some of the most successful models employing hundreds or even thousands of layers. A variety of pathologies such as vanishing exploding gradients make training such deep networks challenging. While residual connections and batch normalization do enable training at these depths, it has remained unclear whether such specialized architecture designs are truly necessary to train deep CNNs. In this work, we demonstrate that it is possible to train vanilla CNNs with ten thousand layers or more simply by using an appropriate initialization scheme. We derive this initialization scheme theoretically by developing a mean field theory for signal propagation and by characterizing the conditions for dynamical isometry, the equilibration of singular values of the input-output Jacobian matrix. These conditions require that the convolution operator be an orthogonal transformation in the sense that it is norm-preserving. We present an algorithm for generating such random initial orthogonal convolution kernels and demonstrate empirically that they enable efficient training of extremely deep architectures.