While many publicly released datasets include an enormous number of labeled examples [1]}, [2]}, [3]}, and recent work on few-shot learning focus on an extreme scenario in which less than one hundred examples are at hand [4]}, [5]}, [6]}, we choose to simulate a broader set of practical scenarios where a small-to-medium dataset is available; in SQuAD [7]}, for example, the average annotation pace is around one minute per question, producing 480 examples in a single 8-hour workday.
Therefore, we consider 4 dataset sizes for each task: 32, 128, 512, and 2048 examples.
For each dataset size, we sample the relevant amount of examples five times using different seeds, thus creating 20 datasets for each task overall.
We report the average score for each dataset size, thereby reducing the high variance associated with training on small datasets.
