Object recognition in real-world requires handling long-tailed or even open-ended data. An ideal visual system needs to reliably recognize the populated visual concepts and meanwhile efficiently learn about emerging new categories with a few training instances. Class-balanced many-shot learning and few-shot learning tackle one side of this problem, via either learning strong classifiers for populated categories or learning to learn few-shot classifiers for the tail classes. In this paper, we investigate the problem of generalized few-shot learning (GFSL) -- a model during the deployment is required to not only learn about "tail" categories with few shots, but simultaneously classify the "head" and "tail" categories. We propose the Classifier Synthesis Learning (CASTLE), a learning framework that learns how to synthesize calibrated few-shot classifiers in addition to the multi-class classifiers of ``head'' classes, leveraging a shared neural dictionary. CASTLE sheds light upon the inductive GFSL through optimizing one clean and effective GFSL learning objective. It demonstrates superior performances than existing GFSL algorithms and strong baselines on MiniImageNet and TieredImageNet data sets. More interestingly, it outperforms previous state-of-the-art methods when evaluated on standard few-shot learning. Visual recognition for objects in the "long tail" has been an important challenge to address (Wang et al., 2017; Liu et al., 2019) . We often have a very limited amount of data on those objects as they are infrequently observed and/or visual exemplars of them are hard to collect. As such, state-of-the-art methods (e.g deep learning) can not be directly applied due to their notorious demand of a large number of annotated data (Krizhevsky et al., 2017; Simonyan & Zisserman, 2014; He et al., 2016) . Few-shot learning (FSL) (Vinyals et al., 2016; Snell et al., 2017; Finn et al., 2017 ) is mindful of the limited instances (i.e, shots) per "tail" concept, which attempts to address this challenging problem by distinguishing between the data-rich "head" categories as SEEN classes and data-scarce "tail" categories as UNSEEN classes. While it is difficult to build classifiers with data from UNSEEN classes, FSL leverages data from SEEN classes to extract inductive biases for effective classifiers acquisition on UNSEEN ones. We refer to (Larochelle, 2018) for an up-to-date survey in few-shot learning. This type of learning, however, creates a chasm in object recognition. Classifiers from many-shot learning for SEEN classes and those from few-shot learning for UNSEEN classes do not mix -they cannot be combined directly to recognize all object categories at the same time. In this paper, we study the problem of Generalized Few-Shot Learning (GFSL), which focuses on the joint classification of both data-rich and data-poor categories. In particular, our goal is for the model trained on the SEEN categories to be capable of incorporating limited UNSEEN class instances, and make predictions for test instances in both the "head" and "tail" of the entire distribution of categories. Figure 1 illustrates the high-level idea of our proposal, contrasting the standard few-shot learning. In contrast to prior works (Hariharan & Girshick, 2017; Wang et al., 2017; Liu et al., 2019 ) that focus on learning "head" and "tail" concepts in a transductive manner, our learning setup requires inductive modeling of the"tail", which is therefore more challenging as we assume no knowledge about the UNSEEN "tail" categories is available during the model learning phase. (GFSL) . GFSL requires to extract inductive bias from SEEN categories to facilitate efficiently learning on few-shot UNSEEN "tail" categories, while maintaining discernability on "head" classes. To this end, we propose Classifier Synthesis Learning (CASTLE), where the few-shot classifiers are synthesized based on a shared neural dictionary across classes. Such synthesized few-shot classifiers are then used together with the many-shot classifiers. To this purpose, we create a scenario, via sampling a set of instances from SEEN categories and pretend that they come from UNSEEN, and apply the synthesized classifiers (based on the instances) as if they are many-shot classifiers to optimize multi-class classification together with the remaining many-shot SEEN classifiers. In other words, we construct few-shot classifiers to not only perform well on the few-shot classes but also to be competitive when used in conjunction with many-shot classifiers of populated classes. We argue that such highly contrastive learning can benefit few-shot classification with high discernibility in its learned visual embeddings (cf. Section 4.2 and Section 4.4). We empirically validate our approach on two standard benchmark data sets -MiniImageNet and TieredImageNet. The proposed approach retains competitive "head" concept recognition performances while outperforming existing approaches on few-shot learning and generalized few-shot learning. We highlight that CASTLE has learned a better calibration between many-shot SEEN classifiers and synthesized UNSEEN classifiers, which naturally addresses the confidence mismatch phenomena , i.e, SEEN and UNSEEN classifiers have different confidence ranges. Building a high-quality visual system usually requires to have a large scale annotated training set with many shots per categories. Many large-scale datasets such as ImageNet have an ample number of instances for popular classes (Russakovsky et al., 2015; Krizhevsky et al., 2017) . However, the data-scarce "tail" of the category distribution matters. For example, a visual search engine needs to deal with the rare object of interests (e.g endangered species) or newly defined items (e.g new smartphone models), which only possess a few data instances. Directly training a system over all classes is prone to over-fit and can be biased towards the data-rich categories. Few-shot learning (FSL) is proposed to tackle this problem, via meta-learning an inductive bias from the SEEN classes, such that it transfers to the learning process of UNSEEN classes with few training data during the model deployment. For example, one line of works uses meta-learned discriminative feature embeddings (Snell et al., 2017; Oreshkin et al., 2018; Rusu et al., 2018; Scott et al., 2018; Ye et al., 2018; Lee et al., 2019) together with non-parametric nearest neighbor classifiers, to recognize novel classes given a few exemplars. Another line of works (Finn et al., 2017; Nichol et al., 2018; Lee & Choi, 2018; Antoniou et al., 2018; Vuorio et al., 2018) chooses to learn a common initialization to a pre-specified model configuration and adapt rapidly using fixed steps of gradient descents over the few-shot training data from UNSEEN categories. FSL emphasizes on building models of the UNSEEN classes and ignore its real-world use case of assisting the many-shot recognition of the "'head" categories. A more realistic setting, i.e, low-shot learning, has been studied before (Hariharan & Girshick, 2017; Wang et al., 2018; Gao et al., 2018; Ye et al., 2018; Liu et al., 2019) . The main aim is to recognize the entire set of concepts in a transductive learning framework -during the training of the target model, you have access to both the SEEN and UNSEEN categories. The key difference to our proposed GFSL is that we assume no access to UNSEEN classes in the learning phase, which requires the model to inductively transfer knowledge from SEEN classes to UNSEEN ones during the evaluation. Previous approaches mostly focus on the transductive setup of GFSL. Some of them (Hariharan & Girshick, 2017; Wang et al., 2018; Gao et al., 2018) apply the exemplar-based classification paradigms on both SEEN and UNSEEN categories to resolve the transductive learning problem. Others (Wang et al., 2017; Schönfeld et al., 2018; Liu et al., 2019) usually ignore the explicit relationship between SEEN and UNSEEN categories, and learn separate classifiers. Ren et al. (2018a) ; Gidaris & Komodakis (2018) propose to solve inductive GFSL via either composing UNSEEN with SEEN classifiers or meta-leaning with recurrent back-propagation procedure. Gidaris & Komodakis (2018) is the most related work to CASTLE, where we differ in how we compose classifiers and the unified learning objective, i.e, we used a learned neural dictionary instead of using MC classifiers as bases. In summary, CASTLE learns both many-shot classifiers and synthesized classifiers via optimizing a single unified objective function, where a classifier composition model with a neural dictionary is leveraged for assembling few-shot classifiers. Our experiments highlight that CASTLE not only outperforms existing methods in terms of GFSL performances from many different aspects, but more interestingly, also improves the classifier's discernibility over standard FSL.
We propose to learn synthesizing few-shot classifiers and many-shot classifiers using one single objective function for GFSL.
{"src": "scitldr-AIC_train", "paper_id": "HJgjuCVKwS"}