Recurrent neural networks are strong dynamic systems, but they are very sensitive to their hyper-parameter configuration. Moreover, training properly a recurrent neural network is a tough task, therefore selecting an appropriate configuration is critical. There have been proposed varied strategies to tackle this issue, however most of them are still impractical because of the time resources needed. In this study, we propose a low computational cost model to evaluate the expected performance of a given architecture based on the distribution of the error of random samples. We validate empirically our proposal using three use case.
Few authors have proposed strategies to address this issue. @cite_14 presented a method for speeding up the hyper-parameter search by detecting and finishing evaluations that under-perform previously computed hyper-parameterizations. Even though they reduced the hyper-parameterization optimization process time down to 50 In contrast to the last approach, we propose and analyze a method to evaluate the hyper-parameters of a given RNN architecture avoiding the high-cost learning (weights optimization) process. Instead of applying a gradient-based optimization procedure to compute the weights, the proposed method uses an approach, i.e., it generates a representative sample of random sets of weights, which are used to compute the error of a given architecture on a test set. This error sampling allow us to reduce the computational cost, cutting down the time of the automatic hyper-parameter methods.
@cite_14
2266822037
Deep neural networks (DNNs) show very strong performance on many machine learning problems, but they are very sensitive to the setting of their hyperparameters. Automated hyperparameter optimization methods have recently been shown to yield settings competitive with those found by human experts, but their widespread adoption is hampered by the fact that they require more computational resources than human experts. Humans have one advantage: when they evaluate a poor hyperparameter setting they can quickly detect (after a few steps of stochastic gradient descent) that the resulting network performs poorly and terminate the corresponding evaluation to save time. In this paper, we mimic the early termination of bad runs using a probabilistic model that extrapolates the performance from the first part of a learning curve. Experiments with a broad range of neural network architectures on various prominent object recognition benchmarks show that our resulting approach speeds up state-of-the-art hyperparameter optimization methods for DNNs roughly twofold, enabling them to find DNN settings that yield better performance than those chosen by human experts.