The models are evaluated with cloze-style questions, whose objective is to fill in the blank to obtain a complete consistent passage.
By selecting the sentence having the highest probability when a candidate sentence substitutes the blank, we can obtain the model answers and assess the performance of SPE.
Experiments are conducted on not only natural language data but also symbolic music data.
On natural language data, the proposed method is validated with synthetic cloze questions as well as the SWAG set [1]} where human crowd workers were involved in question generation.
The experimental results show that our method enhances the performance compared to that based on only word-level CE loss.
