The main drawback of typical word-level LMs, however, is that the captured context depends on the limited length of the previous words, often less than five [1]}, [2]}.
This tendency is prone to limiting the ability of the model in comprehending entire sentence contents and structures [3]}.
To overcome this limitation, sentence-level LMs that directly estimate the probability of an entire sentence without computing the conditional word probabilities have been proposed recently [1]}, [5]}.
These sentence-level LMs have employed noise-contrastive estimation (NCE) that estimates unnormalized density with sampled noise sentences instead of all possible sentences, which are impossible to enumerate.
By learning from the direct supervision of entire sentences, the sentence-level LMs appeared to capture long-term context, leading to improved performance on rescoring the N-best list in ASR.
