This work was supported in part by the Chan Zuckerberg Initiative, in part by IBM Research AI through the AI Horizons Network, and in part by the National Science Foundation under Award No. 1763618. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsor.

Appendix
<FIGURE><FIGURE>Examples of SMLMT and SentCluster can be seen in Figures REF  and REF .
<TABLE>Additional Experiment Results
Results on FewRel 2.0 validation set using the different task distributions is shown in Figure REF .
Full distribution of results in down-stream takss for the various self-supervised tasks can be seen in Fig.  REF

Fine-tuning hyper-parameters
<TABLE>The meta-learning methods learn the learn rate for fine-tuning, thus we only tune the number of steps to run fine-tuning by using development data from 2 tasks (Scitail, Amazon Electronics), following [1]}, [2]}.
We found that running fine-tuning until the loss on the support set is small (\(<=0.01\) ) is an alternative that also performs competitively and does not require tuning the number of steps.
The reported results followed the previous approach and the tuned number of steps of fine-tuning for \(k=8,16,32\)  respectively were: (1) Uniform: 100,75,100 (2) Frequency: 25,150,75 (3) Cluster: 75,50,75 (4) Cluster-ccnet: 150,200,75 (5) SentCluster: 100,250,25 (6) Dynamic: 10, 100, 200. On FewRel we found 20 steps of updates on the support set to perform well on the validation data for all settings.

Other Implementation Details
Since the Fewrel tasks consist of entity pair in the sentence it is important to mark these entities which define the relation to be classified. We used unused tokens in the BERT vocabulary to mark the positions of the entity mentions. Note the in the unsupervised models these unsused tokens get a zero-embedding and are only fine-tuned from the 1-shot or 5-shot support sets.
<FIGURE>Hyper-parameters for meta-training are listed in Table REF . Dataset statistics for downstream classification tasks can be found in [1]} and few-shot splits can be downloaded from https://github.com/iesl/leopard.
Training Hardware:
The models were trained on 32 V100 GPU. Training takes about 42 hours.
