We use derivatives of a word-level recurrent neural network (RNN) LM [1]} for implementation of the estimation model of data distribution.
With the simple neural LM, this study attempts to focus on proposing a training objective rather than devising a novel neural architecture for language modeling.
In noise sampling, we suggest two strategies, including batch NCE [2]} and editing original sentences with a pre-trained bidirectional LM [3]} to obtain realistic fake sentences.
