The other method, referred to as sentence resampling in this paper, uses a pre-trained word-level bidirectional RNN to resample transformed sentences from original sentences.
To sample a new sentence replaced with multiple new words, we adopt the masked LM training style [1]} in which about 15% of input word tokens are replaced with the <MASK> token, and then the model learns to reconstruct the original word sequences.
Then, the resampling procedure for the noise sampling uses the following operations: input masking, predicting the sequence of word probabilities (identically to the masked LM training so far), and sampling words at the masked positions with the predicted probabilities.
A bidirectional RNN is optimized in advance for the NCE training during which the RNN is not updated.
We remark that we set the network structure of the bidirectional RNN same as that of [2]}, which predicts the target word based on both preceding words and following words, except for tying forward embeddings, backward embeddings, and output embeddings.
