Broad Discourse Context for Language Modeling
Recurrent neural network language models (RNNLM) have become a key element for state of the art language modeling and are integral parts in a wide range of applications such as speech recognition or information retrieval systems. However, recent work has shown that the extent to which these models truly understand language remains unclear. In particular, the LAMBADA dataset consists of a collection of hard word prediction examples specifically designed to test language models' capabilities to keep track of information in the broader discourse. In this thesis we delve into the nature of LAMBADA and highlight the impact that rare words have on the overall results. We introduce our proposed model, the Softmax Mixture, and showcase through a series of experiments its ability to handle rare words when compared to vanilla RNNLMs. We further implement the Pointer Sentinel Mixture model and demonstrate that it achieves perplexities on par with the state ofthe art on LAMBADA while having superior adaptation capabilities.