We prepared the SWAG [1]} questions, each of which contains an incomplete passage consisting of a preceding sentence followed by a blank sentence, and multiple candidate sentences for the blank.
To provide more evaluation results on diverse datasets, we also automatically generate cloze-style questions that have the same format as that of the original SWAG questions of which crowd workers verified the validity.
We refer to this question format as sentence-cloze hereafter.
For the synthetic question generation, we reused the batch NCE and the sentence resampling (Section REF ).
In the batch NCE case, the candidate sentences of validation questions were kept from being presented in the training phase by partitioning a corpus.
While the synthetic questions are much easier than the SWAG questions since the training and validation tasks are aligned, it provides the basic proof-of-concept of the proposed sentence-level NCE training.
