1 Henceforth we will use terms like "RNN" and "LSTM" with the understanding that we are referring to language models that use these formalisms have outperformed their count-based counterparts <cite>(Chelba et al., 2013</cite>; Zaremba et al., 2014; Mikolov, 2012) .