SMLMT proposes sentence-level tasks and thus lacks cross-sentence reasoning.
This is confirmed by the poor downstream few-shot performance of models trained on SMLMT (see Sec. REF ).
Since models trained on SMLMT have never seen pairs of sentences as input, it leads to a train-test mismatch for sentence-pair classification tasks.
To remedy this, we introduce a simple but effective contrastive learning task over sentence-pairs that bridges this gap.
Contrastive learning has been used to learn effective sentence representations [1]}.
Next sentence prediction, a sentence-pair task, was used in the training of BERT [2]} which was later found to be not effective [3]}.
BERT considered segments instead of full sentences, however the downstream tasks often require reasoning over complete sentences.
Thus, we consider classifying whether two sentences come from the same document as opposed to different documents, as a sentence-pair task to enable cross-sentence reasoning.
This simple objective was found to be quite effective in our experiments.
Note that during meta-training, this can be treated as an additional task in the task distribution. Since the SMLMT task distribution consists of an exponential number of tasks, we sample the sentence-pair task in an episode with a fixed probability \(\alpha \) , which is hyper-parameter.
