Maxout neurons for deep convolutional and LSTM neural networks in speech recognition
We combine maxout neurons with convolutional and LSTM structures for DNNs.The optimal network structures and training strategies are explored for the models.Experiments are carried out for 6 languages on the IARPA Babel data sets.State of the art results are achieved after system combination. Deep neural networks (DNNs) have achieved great success in acoustic modeling for speech recognition. However, DNNs with sigmoid neurons may suffer from the vanishing gradient problem during training. Maxout neurons are promising alternatives to sigmoid neurons. The activation of a maxout neuron is obtained by selecting the maximum value within a local region, which results in constant gradients during the training process. In this paper, we combine the maxout neurons with two popular DNN structures for acoustic modeling, namely the convolutional neural network (CNN) and the long short term memory (LSTM) recurrent neural network (RNN) The optimal network structures and training strategies for the models are explored. Experiments are conducted on the benchmark data sets released under the IARPA Babel Program. The proposed models achieve 2.5 6.0% relative improvements over their corresponding CNN or LSTM RNN baselines across six language collections. The state of the art results on these data sets are achieved after system combination.
Speech Commun.