We propose a large margin criterion for training neural language models. Conventionally, neural language models are trained by minimizing perplexity (PPL) on grammatical sentences. However, we demonstrate that PPL may not be the best metric to optimize in some tasks, and further propose a large margin formulation. The proposed method aims to enlarge the margin between the "good" and "bad" sentences in a task-specific sense. It is trained end-to-end and can be widely applied to tasks that involve re-scoring of generated text. Compared with minimum-PPL training, our method gains up to 1.1 WER reduction for speech recognition and 1.0 BLEU increase for machine translation.
While these works all adopt PPL as the metric to optimize, sometimes one may optimize a task-specific objective. For example, and propose discriminative LMs to improve speech recognition. The common methodology therein is to fit a probabilistic model, e.g., conditional random field @cite_10 , to the space of text candidates, and maximize the probability at the desired candidate. The problem is often solved by perceptron algorithm. However, these methods all rely on ad-hoc choice of features, e.g., counts of @math -grams where @math varies in a small range (e.g.,1 to 3). Moreover, it is also not clear how these methods would take advantage of an existing language model (trained on large unsupervised corpus). Nevertheless, the same methodology can be extended to RNNLMs, thus avoiding the aforementioned limitations. For example, train an RNNLM by favoring sentences with high BLEU scores and integrate it into a phrase-based MT decoder.
@cite_10
2158025800
Recurrent neural network language models (RNNLMs) have recently shown to outperform the venerable n-gram language models (LMs). However, in automatic speech recognition (ASR), RNNLMs were not yet used to directly decode a speech signal. Instead, RNNLMs are rather applied to rescore N-best lists generated from word lattices. To use RNNLMs in earlier stages of the speech recognition, our work proposes to transform RNNLMs into weighted finite state transducers approximating their underlying probability distribution. While the main idea consists in discretizing continuous representations of word histories, we present a first implementation of the approach using clustering techniques and entropy-based pruning. Achieved experimental results on LM perplexity and on ASR word error rates are encouraging since the performance of the discretized RNNLMs is comparable to the one of n-gram LMs.