In the supervised setting, in particular, meta-learning task distribution is often defined by sub-sampling from the classes in a classification problem over a fixed dataset [1]}.
This not only limits the applicability of meta-learning to the underlying classification problem,
but also requires a diverse set of supervised datasets with a large number of classes to enable learning.
Self-supervised meta-learning, on the other hand, seeks to propose tasks from unlabelled data [2]}, [3]},
and has great potential to enable numerous important applications [4]} such as neural architecture search, continual learning, hyper-parameter optimization, learning in low-resource settings, etc.
Existing work in meta-learning for NLP, however, defaults to task distributions that tend to be overly simplistic, e.g. using existing supervised datasets [5]}, [6]}, [7]} or unsupervised cloze-style tasks with uniform selection of words from the vocabulary [3]}.
Given the lack of exploration on this critical component, we propose to devise and evaluate various task distributions in the context of unsupervised meta-learning for NLP.
