Recurrent neural networks are strong dynamic systems, but they are very sensitive to their hyper-parameter configuration. Moreover, training properly a recurrent neural network is a tough task, therefore selecting an appropriate configuration is critical. There have been proposed varied strategies to tackle this issue, however most of them are still impractical because of the time resources needed. In this study, we propose a low computational cost model to evaluate the expected performance of a given architecture based on the distribution of the error of random samples. We validate empirically our proposal using three use case.
In general, RNNs are difficult to train by gradient-based optimization procedures, e.g., stochastic gradient descent or second-order methods, due to the exploding and the vanishing gradient problems @cite_16 @cite_24 @cite_1 .
@cite_24 @cite_16 @cite_1
1899504021 2107878631 1815076433
This chapter contains sections titled: Introduction Exponential Error Decay Dilemma: Avoiding Aradient Decay Prevents Long-Term Latching Remedies Conclusion ]]> Recurrent neural networks can be used to map input sequences to output sequences, such as for recognition, production or prediction problems. However, practical difficulties have been reported in training recurrent neural networks to perform tasks in which the temporal contingencies present in the input output sequences span long intervals. We show why gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases. These results expose a trade-off between efficient learning by gradient descent and latching on information for long periods. Based on an understanding of this problem, alternatives to standard gradient descent are considered. > There are two widely known issues with properly training recurrent neural networks, the vanishing and the exploding gradient problems detailed in (1994). In this paper we attempt to improve the understanding of the underlying issues by exploring these problems from an analytical, a geometric and a dynamical systems perspective. Our analysis is used to justify a simple yet effective solution. We propose a gradient norm clipping strategy to deal with exploding gradients and a soft constraint for the vanishing gradients problem. We validate empirically our hypothesis and proposed solutions in the experimental section.