A word-level RNN LM [1]} is a popular deep neural LM that benefits from the strength of RNN in dealing with sequential data.
The network takes as input a word each time step and then outputs the estimation of the probability distribution for the next word.
The estimation of the joint probability over an entire sentence can be naturally obtained by the product of those conditional probabilities.
Due to its popularity and prominence in NLP, the word-level RNN LM has been compared as a baseline model for a variety of NLU tasks [2]}, [3]}.
Network architectural modification, including replacing or combining recurrent layers with convolutional layers [4]}, [5]}, [6]}, inserting attention mechanism [7]}, or adopting a transformer-based LM [8]}, [9]} can be applied together with our main idea, and it is left for future research.
