Class imbalance at the meta-dataset level is caused by imbalanced dataset classes in one (or more) of the three data splits: meta-training, meta-validation, meta-testing. This disproportion affects the distribution of tasks that a model is exposed to during meta-training, affecting their ability to generalize to new tasks. FSL methods developed for benchmark meta-dataset (e.g. Mini-ImageNet) assume a balanced set of classes during meta-training (e.g. MAML) and the representation learning phase (e.g. Baseline++). However, deploying these methods in the real-world will likely result in combining several meta-datasets into a single meta-learning corpus [1]} or being swapped for domain-specific meta-datasets [2]}, [3]}, which is likely to result in a possibly large or even unknown imbalance distribution. In this work, we propose to loosen the assumption of a balanced meta-dataset, and investigate a more practical question: how does imbalance impact meta-learning of FSL methods? While it is generally understood that class imbalance harms representation learning [4]}, little is understood about how it impacts meta-learning ability.
Standard assumptions in the study of long-tail distributions [4]} are immediate data availability, and distribution according to Zipf/Power Law [6]}.
Here, we propose a more realistic setting, supposing that data is not available at once and that the underlying class distribution is unknown. This is the case in many real-world settings (e.g. online class imbalance, [7]}). Therefore, our study answers another vital question: what happens if we have no knowledge of the underlying data distribution but still sample classes uniformly at random?
<FIGURE>