Investigation on log linear interpolation of multi domain neural network language model
Inspired by the success of multi task training in acoustic modeling, this paper investigates a new architecture for a multi domain neural network based language model (NNLM) The proposed model has several shared hidden layers and domain specific output layers. As will be shown, the log linear interpolation of the multi domain outputs and the optimization of interpolation weights fit naturally in the framework of NNLM. The resulting model can be expressed as a single NNLM. As an initial study of such an architecture, this paper focuses on deep feed forward neural networks (DNNs) We also re investigate the potential of long context up to 30 grams, and depth up to 5 hidden layers in DNN LM. Our final feed forward multidomain NNLM is trained on 3.1B running words across 11 domains for English broadcast news and conversations large vocabulary continuous speech recognition task. After log linear interpolation and fine tuning, we measured improvements in terms of perplexity and word error rate over the models trained on 50M running words of in domain news resources. The final multi domain feed forward LM outperformed our previous best LSTM RNN LM trained on the 50M in domain corpus, even after linear interpolation with large count models.
2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)