SMLMT uses a data-augmentation strategy to automatically assign labels to unlabelled sentences by consistently masking out the same word type in a set of sentences. The masked out word then serves as the label for the sentences.
This cloze-style approach to creating tasks was inspired by the success of masked language modeling [1]} in learning useful representations.
While this leads to significant improvements in sentence-level classification on a range of real downstream tasks [2]},
it is unclear whether a word masking approach is the most efficient to learning useful sentence representations.
To probe this question further, we explore an alternative to SMLMT that directly assigns semantic labels to sentences without any augmentation.
