Evolutionary optimization of long short term memory neural network language model
Recurrent neural network language models (RNN LMs) are recently proven to produce better performance than conventional N gram based language models in various speech recognition tasks. Especially, long short term memory recurrent neural network language models (LSTM LMs) give superior performance for its ability to better modeling word history information. However, LSTM LMs have complex network structure and training configurations, which are meta parameters that need to be well tuned to achieve the state of the art performance. The tuning is usually performed manually by humans, but it is not easy because it requires expert knowledge and intensive effort with many trials. In this study, we apply covariance matrix adaptation evolution strategy (CMA ES) and automate the tuning process. CMA ES is one of the most efficient global optimization techniques that has demonstrated superior performance in various benchmark tasks. In the experiments, the meta parameters subject to the tuning included unit types at e.