We consider all the different approaches to self-supervised task distributions described in Sec  and the baseline approach of SMLMT:
(1) Uniform: this is the SMLMT approach of [1]} which use uniform random sampling over word-types;
(2) Frequency: SMLMT with a sampling proportional to log-frequency (see REF );
(3) Cluster: SMLMT where labels are picked from same word cluster (see REF );
(4) Dynamic: curriculum-based task sampling with Cluster as the static distribution (see REF );
(5) Cluster-ccnet: same as Cluster but using ccnet [2]} as the corpora, which consists of web crawled data;
(6) SentCluster: alternative to SMLMT which proposes tasks from subsets of sentence clustering (see REF );
(7) SentPair: the sentence-pair tasks (see REF ).
All methods, except SentCluster and Cluster-ccnet, have Wikipedia as the text corpora. The sentence embeddings for SentCluster task distribution were obtained from [3]}, and consist of embeddings of about 1 billion sentences from ccnet [2]}. For this reason, we also report Cluster-ccnet that uses this same set of sentences.
We found it beneficial to include 25% Frequency tasks in the Cluster task distribution and SentPair tasks are included in all other task distributions unless otherwise noted.
Note that we only consider completely unsupervised meta-learning methods for fair evaluation.
However our results improve over [1]} which showed improvements over BERT and multi-task BERT baselines. As we utilize the same dataset splits released in their work, our results can be directly compared.
