We argue that the widely used Omniglot and miniImageNet benchmarks are too simple because their class semantics do not vary across episodes, which defeats their intended purpose of evaluating few-shot classification methods. The class semantics of Omniglot is invariably “characters” and the class semantics of miniImageNet, “object category”. Because the class semantics are so similar, we propose a new method called Centroid Networks which can achieve surprisingly high accuracies on Omniglot and miniImageNet without using any labels at metaevaluation time. Our results suggest that those benchmarks are not adapted for supervised few-shot classification since the supervision itself is not necessary during meta-evaluation. The Meta-Dataset, a collection of 10 datasets, was recently proposed as a harder few-shot classification benchmark. Using our method, we derive a new metric, the Class Semantics Consistency Criterion, and use it to quantify the difficulty of Meta-Dataset. Finally, under some restrictive assumptions, we show that Centroid Networks is faster and more accurate than a state-of-the-art learning-to-cluster method (Hsu et al., 2018). Supervised few-shot classification, sometimes simply called few-shot learning, consists in learning a classifier from a small number of examples. Being able to quickly learn new classes from a small number of labeled examples is desirable from a practical perspective because it removes the need to label large datasets. Typically, supervised few-shot classification is formulated as meta-learning on episodes, where each episode corresponds to two small sets of labeled examples called support and query sets. The goal is to train a classifier on the support set and to classify the query set with maximum accuracy. The Omniglot (Lake et al., 2011) and miniImageNet (Vinyals et al., 2016; Ravi & Larochelle, 2017) benchmarks have been heavily used to evaluate and compare supervised few-shot classification methods in the last few years (Vinyals et al., 2016; Ravi & Larochelle, 2017; Snell et al., 2017; Finn et al., 2017; Sung et al., 2018) . Despite their popularity and their important role in pioneering the few-shot learning field, we argue that the Omniglot and miniImageNet benchmarks should not be taken as gold standards for evaluating supervised few-shot classification because they rely on consistent class semantics across episodes. Specifically, Omniglot classes always correspond to alphabet characters, while miniImageNet classes always correspond to object categories as defined by the WordNet taxonomy (Miller, 1995; Russakovsky et al., 2015) . One consequence is that benchmarks with consistent class semantics have similar class semantics between meta-training and meta-evaluation 1 . Therefore, they are too "easy" because they do not test the ability of supervised few-shot classification methods to adapt to new class semantics. From an applications perspective, being able to adapt to changing class semantics is a desirable feature. For instance, if the application is to organize users' personal photo gallery, different users might want to sort their personal photo gallery according to the different semantics, such as person identity, place or time. From a methodological perspective, we argue that supervised few-shot classification becomes an awkward task in the ideal case where the class semantics are perfectly consistent. Indeed, if the end goal of every episode is to classify the query set according to the same class semantics, do we even need the support set to define the classes, once the semantics are learned ? Consider the characters below, extracted from the "Mongolian" alphabet of Omniglot. How would you group the characters below? This task is not particularly hard, even if the reader was never shown labeled examples prior to the task, simply because the reader was already familiar with the class semantics of interest (characters), and can generalize them to new classes. This simple observation suggests that when class semantics are consistent, few-shot learning algorithms might not actually need labels during metaevaluation. To show this, we introduce a new learning-to-cluster 2 method called Centroid Networks which achieves surprisingly high accuracies on Omniglot and miniImageNet without using any labels at meta-evaluation time. 3 The method is very similar to Prototypical Networks (Snell et al., 2017) , but the key difference is that the labels of the support set can be reliably recovered through clustering whenever the cluster semantics are consistent across tasks. A harder benchmark would involve selecting different cluster semantics across episodes. For example, consider the following set of shapes: In this case, the task remains ambiguous because clustering semantics (e.g. shape, color, border style) have not been specified. To classify such a set requires either supervision, such as a labeled support set, or to somehow know the class semantics beforehand. Following that spirit, the Meta-Dataset, a collection of 10 datasets, was recently proposed as a harder and more realistic few-shot classification benchmark (Triantafillou et al., 2019) . Among other things such as variable numbers of ways and shots, a key difficulty of the Meta-Dataset is that class semantics vary across episodes, since episodes are generated from a randomly selected dataset. We propose to use Centroid Networks to benchmark how hard this dataset is. In particular, we suggest looking at the gap between the performance of Prototypical Networks and Centroid Networks, which we call the class semantics consistency criterion (CSCC). We proposed Centroid Networks for performing clustering without labels at meta-evaluation time, and with the idea of using it to assess the difficulty of few-shot classification benchmarks. First, we validate our method by beating a state-of-the-art few-shot clustering method (Hsu et al., 2018) in the setting of a known number of equally-sized clusters, with the advantage that our method is easier to train and orders of magnitude faster to run. Then, we define the CSCC metric from the unsupervised accuracy of Centroid Networks, and use it for quantifying the difficulty of current few-shot learning benchmarks in terms of class semantics consistency. We find that Omniglot has extremely consistent class semantics (CSCC close to 1), and that miniImageNet has fairly high CSCC as well (CSCC close to 0.8), which backs the intuition that its class semantics invariably correspond to object categories. Our results on the Meta-Dataset benchmark show that it has much lower CSCCs than Omniglot in all settings, and lower CSCCs than miniImageNet in the ILSVRC only setting, which confirms that Meta-Dataset has harder and more diverse class semantics. As future work, we would like to improve the CSCC by making it more interpretable and less dependent on the backbone architectures. A APPENDIX : BACKGROUND AND IMPLEMENTATION DETAILS A.1 SINKHORN DISTANCES The Wasserstein-2 distance is a distance between two probability masses p and q. Given a base distance d(x, x ), we define the cost of transporting one unit of mass from x to x as d(x, x ) 2 . The Wasserstein-2 distance is defined as the cheapest cost for transporting all mass from p to q. When the transportation plan is regularized to have large entropy, we obtain Sinkhorn distances, which can be computed very efficiently for discrete distributions (Cuturi, 2013; Cuturi & Doucet, 2014) (entropy-regularization makes the problem strongly convex). Sinkhorn distances are the basis of the Sinkhorn K-Means algorithm, which is the main component of Centroid Networks. In Algorithm 1, we describe the Sinkhorn algorithm in the particular case where we want to transport mass from the weighted data points (x i , R j ) to the weighted centroids (c j , C j ), where R j and C j are the weights of the data points and centroids, respectively. In practice, we leverage the log-sum-exp trick in the to avoid numerical underflows. A.2 DATA SPLITS AND ARCHITECTURE FOR OMNIGLOT AND miniIMAGENET EXPERIMENTS For the embedding network for the Omniglot and miniImageNet, we reuse exactly the same simple convolutional architecture as in Prototypical Networks (Snell et al., 2017) , which consists of four stacked blocks (2D convolution with 3 × 3 kernel and stride 1, BatchNorm, ReLU, and 2 × 2 max-pooling), the output of which is flattened. This results in a 64-dimensional embedding for Omniglot and 1600-dimensional embedding for miniImageNet. For miniImageNet, we pretrain the embedding function using prototypical networks to solve 30-way problems instead of 5, which is the recommended trick in the paper (Snell et al., 2017) . For the other settings, we train from scratch. Omniglot (Lake et al., 2011) consists of a total of 1623 classes of handwritten characters from 50 alphabets, with 20 examples per class. Images are grayscale with size 28 × 28. We follow the same protocol as in Prototypical Networks and use the "Vinyals" train/validation/test splits. We consider 5-way 5-shot and 20-way 5-shot settings (15 query points per class). miniImageNet (Vinyals et al., 2016) consists of 100 classes, each containing 600 color images of size 84 × 84. We follow the "Ravi" splits: 64 classes for training, 16 for validation, and 20 for testing. We consider the 5-way 5-shot setting (15 query points per class).
Omniglot and miniImageNet are too simple for few-shot learning because we can solve them without using labels during meta-evaluation, as demonstrated with a method called centroid networks
{"src": "scitldr-AIC_train", "paper_id": "SygeY1SYvr"}