Neural language models (NLMs) are generative, and they model the distribution of grammatical sentences. Trained on huge corpus, NLMs are pushing the limit of modeling accuracy. Besides, they have also been applied to supervised learning tasks that decode text, e.g., automatic speech recognition (ASR). By re-scoring the n-best list, NLM can select grammatically more correct candidate among the list, and significantly reduce word/char error rate. However, the generative nature of NLM may not guarantee a discrimination between “good” and “bad” (in a task-specific sense) sentences, resulting in suboptimal performance. This work proposes an approach to adapt a generative NLM to a discriminative one. Different from the commonly used maximum likelihood objective, the proposed method aims at enlarging the margin between the “good” and “bad” sentences. It is trained end-to-end and can be widely applied to tasks that involve the re-scoring of the decoded text. Significant gains are observed in both ASR and statistical machine translation (SMT) tasks. Language models (LMs) estimate the likelihood of a symbol sequence {s i } n i=0 , based on the joint probability, p(s 0 , . . . , s n ) = p(s 0 ) n i=1 p(s i |s i−1 , s i−2 , . . . , s 0 ).(1 ) works, BID7 and BID25 , propose to predict the next symbol based on a fusion of the hidden states in the ASR/SMT and language model. A gating mechanism is jointly trained to determine how much the language model should contribute.The afore-discussed language models are generative in the sense that they merely model the joint distribution of a symbol sequence (Eq. ( 1) ). While the research community is mostly focused on pushing the limit of modeling accuracy (lower PPL) (e.g., BID12 , very limited attention has been paid to the discrimination ability of language models when they are applied to supervised learning tasks, such as ASR and SMT. Discriminative language modeling aims at enhancing the performance in supervised learning tasks. In specific, existing works BID23 BID10 BID21 often target at improving ASR accuracy. The key motivation underlying them is that the model should be able to discriminate between "good" and "bad" sentences in a task-specific sense, instead of just modeling grammatical ones. The common methodology is to build a binary classifier upon hand-crafted features extracted from the sentences. However, it is not obvious how these methods can utilize large unsupervised corpus, which is often easily available, and the hand-crafted features are also ad hoc and may result in suboptimal performance.In this work, we study how to improve the discrimination ability of a neural language model. The proposed method enlarges the difference between the log-likelihoods of "good" and "bad" sentences. In contrast to the existing works BID23 BID10 BID21 , our method does not rely on hand-crafted features. It is trained in end-to-end manner and able to take advantage of large external text corpus. We apply the proposed large margin language model to ASR and SMT tasks. It reduces word error rate (WER) and increases bilingual evaluation understudy (BLEU) scores significantly, showing notable advantage over several alternative methods that are well adopted.2 RELATED WORK BID23 BID10 and BID21 proposed to train discriminative language models based on hand crafted features. They essentially build linear classifiers that give high scores on "good" sentences but low scores on "bad" ones. These methods all rely on ad hoc choice of features, e.g., counts of n-grams where n varies in a small range (e.g., 1 ∼ 3). Moreover, it is also not clear how these methods would take advantage of an existing language model (trained on large unsupervised corpus). BID28 tries to overcome the above issues by adapting an NLM on the transcriptions of a speech dataset. Although the setup is more similar to ours, their objective is not well-behaved and difficult to optimize when there are multiple beam candidates. An in-depth discussion will be given in Section 3.1. BID15 designed another approach to train a discriminative language model, which is based on bi-grams. Similar to our method, the objective there aims at increasing the difference between the scores of the best candidate and ground-truth. However, since the language model is not end-to-end, there are several issues complicating the training, e.g., handling back-off weight.Our proposed method is based on comparisons between pairs of sentences. Its implementation resembles siamese network architecture BID3 BID29 , first proposed for face verification tasks. Recently, siamese network has also been applied to learning similarities on sequences BID19 BID20 . In spite of solving different problems, the common methodology is to extract a pair of hidden representations for a pair of input samples (through a shared network). It then manipulates the distance between the hidden representations based on whether the two samples are considered similar or not. Our work also draws some inspirations from information retrieval (IR) BID16 . As a representative IR method, ranking SVM BID8 assumes a linear scoring function, and imposes a hinge loss on the difference between the scores of sample pairs. Conventional language models are guided by minimizing perplexity, and they are generative models. This work proposes an approach to enhance the discrimination ability of language models. It is trained end-to-end by maximizing the margin between "good" and "bad" (in a task-specific sense) sentences. The method is general and can be applied to various tasks that require re-scoring of text data. Experiments on ASR and SMT have shown a consistent gain over several baselines. These facts argue that min-perplexity is not necessarily an appropriate guideline when we want to apply language models in some supervised learning problems. A future direction is to apply the proposed method to conversation generation. The goal is to discriminate between boring (e.g., "I don't know") and informative replies, thus deprecating the former. Another interesting future work is to apply the LMLM/rank-LMLM to lattices during decoding.
Enhance the language model for supervised learning task 
{"src": "scitldr-AIC_train", "paper_id": "H1g-gk5EuQ"}