Recurrent neural networks are strong dynamic systems, but they are very sensitive to their hyper-parameter configuration. Moreover, training properly a recurrent neural network is a tough task, therefore selecting an appropriate configuration is critical. There have been proposed varied strategies to tackle this issue, however most of them are still impractical because of the time resources needed. In this study, we propose a low computational cost model to evaluate the expected performance of a given architecture based on the distribution of the error of random samples. We validate empirically our proposal using three use case.
It has been reported numerous times in the neural network research literature that weight initialization in neural networks affects the learning rate, the convergence rate and the probability of correct classification. In this research paper we develop a theory for objectively testing various weight initialization strategies. Our theory provides a quantitative measure for each available weight initialization strategy. Thus for each initialization strategy and each epoch we estimate the conditional probability distribution function of correct classification given the epoch number. For each initialization strategy and for a given epoch the conditional probability is a random variable with certain probability distribution function and certain mean and variance. Based on multivariate analysis, statistics of extremes, analysis of variance and estimation theory we develop an objective framework and measurements to assess if one strategy is better than another or if the differences between strategies are not significant but they are due to random fluctuations.