The results confirm that the proposed method improved the accuracy of the word-level RNN on the original SWAG questions, even though it did not exploit the negative sentences humans provided.
In addition, the performance gaps between when using the relaxed softmax and when using the vanilla softmax for the original SWAG questions were larger than those for the synthetic sentence-cloze ones.
On the other hand, the performance of the ESIM was not as good as reported in [1]}, which can be explained by the facts that any pre-trained word embeddings learned from external data were not used and that the NCE training did not exploit the human-supervised negative sentences.
Another interesting observation is that the performance of the proposed method did not vary a lot between the batch NCE and the sentence resampling, while on the contrary, ESIM performed poorly when trained with the sentence resampling.
This implies to some extent that SPE with the NCE training is robust to how training questions are generated.
