Training recurrent neural networks (RNNs) that possess long-term memory is challenging. We provide insight into the trade-off between the smoothness of the cost function and the memory retention capabilities of the network. We express both aspects in terms of the Lipschitz constant of the dynamics modeled by the network. This allows us to make a distinction between three types of regions in the parameter space. In the first region, the network experiences problems in retaining long-term information, while at the same time the cost function is smooth and easy for gradient descent to navigate in. In the second region, the amount of stored information increases with time and the cost function is intricate and full of local minima. The third region is in between the two other regions and here the RNN is able to retain long-term information. Based on these theoretical findings we present the hypothesis that good parameter choices for the RNN are located in between the well-behaved and the ill-behaved cost function regions. The concepts presented in the paper are illustrated by artificially generated and real examples.
conditions for exploding gradients to appear are also provided in @cite_1 . Their result can be understood as a special case of Theorem and is limited to a single layer of vanilla RNN. By constraining the results to this vanilla case, they do not cover interesting nonlinear behaviors, such as periodic behavior and strange attractors, that are studied here. Our results are also more general, because we include bounds on second order derivatives.
@cite_1
2783402179
We give a rigorous analysis of the statistical behavior of gradients in randomly initialized feed-forward networks with ReLU activations. Our results show that a fully connected depth @math ReLU net with hidden layer widths @math will have exploding and vanishing gradients if and only if @math is large. The point of view of this article is that whether a given neural net will have exploding vanishing gradients is a function mainly of the architecture of the net, and hence can be tested at initialization. Our results imply that a fully connected network that produces manageable gradients at initialization must have many hidden layers that are about as wide as the network is deep. This work is related to the mean field theory approach to random neural nets. From this point of view, we give a rigorous computation of the @math corrections to the propagation of gradients at the so-called edge of chaos.