Few-shot Open-set Recognition Using Background as Unknowns

Few-shot open-set recognition aims to classify both seen and novel images given only limited training data of seen classes. The challenge of this task is that the model is required not only to learn a discriminative classifier to classify the pre-defined classes with few training data but also to reject inputs from unseen classes that never appear at training time. In this paper, we propose to solve the problem from two novel aspects. First, instead of learning the decision boundaries between seen classes, as is done in standard close-set classification, we reserve space for unseen classes, such that images located in these areas are recognized as the unseen classes. Second, to effectively learn such decision boundaries, we propose to utilize the background features from seen classes. As these background regions do not significantly contribute to the decision of close-set classification, it is natural to use them as the pseudo unseen classes for classifier learning. Our extensive experiments show that our proposed method not only outperforms multiple baselines but also sets new state-of-the-art results on three popular benchmarks, namely tieredImageNet, miniImageNet, and Caltech-USCD Birds-200-2011 (CUB).


INTRODUCTION
Deep Neural Networks have made great success in computer vision tasks, such as image recognition, image segmentation, text understanding, and object detection [6,20,24,52,56]. Deep models based on Convolutional Neural Network (CNN) [11,13,17,35] architectures have been proved efficient in excavating representative information from data in many fields.
Supervised learning is one of the most dominant learning approaches for deep models, where the optimization of deep models can be achieved by feeding a large amount of training data to the models and utilizing their labels as the training supervision. However, supervised learning limits the model application to the pre-defined training classes, which is often described as the closeset learning task. It is problematic to directly deploy these models to the real world, as a variety of novel categories that are out of the training classes may exist.
As it is impossible to enumerate all the real-world classes for training, the learned model has to recognize inputs from new classes as one of the known classes, as illustrated in Fig. 1 (a), which hampers the applications of deep models in the real world.
Open-set recognition (OSR) is proposed, as a promising direction, to solve this kind of unknown class rejection problem. It aims to accurately classify the known classes and meanwhile detect the unknown classes. Many OSR methods make use of the large amount of training data to estimate the distributions of training classes, such that those out-of-distribution data can be recognized as unseen classes.
This has largely solved the open-set recognition problem in many tasks. However, there are still many application scenarios where these algorithms are not applicable. For example, in longtailed recognition, when the training data of some classes, e.g., rare animals, is not fairly enough, it is difficult to depict the distributions of these classes. Moreover, for tasks that rely on limited data to make predictions, such as few-shot learning, face verification, and person re-identification, these distribution-based methods would also fail.
To handle the problems in these special settings, a more challenging task, namely, Few-shot Open-Set Recognition (FSOSR) is proposed [21], which aims to classify inputs from both known classes and unknown classes with only a few labeled training data of seen classes available.
The goal of FSOSR model is twofold: to identify discriminative information from limited training data, and to have the ability to recognize unseen samples. The well-studied few-shot learning literature has well addressed the former problem, hence the main challenge of FSOSR is to detect unseen samples without harming the classification accuracy, which is the focus of this paper. The main difficulty is that the information from training data to use for obtaining discriminative classifiers of seen classes is already very limited. A question is then raised -where can we find the information of unseen classes? In this paper, we argue that the background regions provide valuable information that can be utilized to detect samples from novel classes. In closed-set recognition, a CNN is optimized to identify the discriminative information in the training samples, with the help of activation functions. As a result, only the discriminative parts in the images are kept as the representations of the data for classification, and the backgrounds are therefore ignored. Since these ignored backgrounds do not belong to any of these existing classes, we treat them as the pseudo unseen classes to learn a classifier that can classify both seen and unseen classes meanwhile. Instead of learning the decision boundaries of only seen classes, we reserve the space for unseen classes with the representations from the background, and we can easily conduct open-set classification by detecting samples located in these areas. Our motivation is illustrated in Fig. 1 (b). More specifically, we first initialize a class-mean classifier, where the prototypes, i.e., the weight vectors corresponding to different classes, are obtained by averaging embeddings of support samples from each class. We then randomly initialize a set of prototypes that correspond to unseen classes. Next, we use the background features as well as the features of support data to finetune the whole classifier. The labels of the background features are set as the nearest prototypes that correspond to unseen classes.
To accurately locate the background regions given limited training data, we further develop a Progressive Class Activation Mapping (ProCAM) module. Compared with the standard CAM [57], the Pro-CAM iteratively mines the class-related regions in images, which can better separate the foreground and background areas in the low-show case as shown in Fig. 2. We empirically show that our proposed ProCAM can not only generate better prototypes of unseen classes, but also regularize the learning of close-set classification.
Extensive experiments on multiple benchmark datasets have been conducted to verify the effectiveness of our proposed method. Our main contributions in this work can be summarized as follows.
• We solve the few-shot open-set recognition task as a multiclass classification problem, where a set of additional classes is added as unseen classes. • To obtain the data used for training the classifier, we propose to utilize the background regions of images from the seen classes. • We develop a Progressive CAM (ProCAM) module to iteratively seperate the class-related foreground regions and the background regions. • Experiments on the tieredImageNet, CUB, and miniImageNet datasets show that our method significantly outperforms the baselines and sets new state-of-the-art performance with remarkable advantages.

RELATED WORK 2.1 Few-shot learning
Few-shot learning focuses on training models with insufficient training samples to gain the ability of classifying unseen images [5,12,23,25,36,37,41,45,47,[49][50][51][53][54][55]. There are great diversities shown up in the literature of few-shot learning. The two main streams in the few-shot learning literature are optimization-based methods [4,7,18,32,34,37] and metric-based methods [12,36,43,46,47,50]. Optimization-based methods target at finding good initial parameters and adapting these parameters to new tasks effectively with low shot supports. Ravi and Larochelle [32] optimize the model parameters using LSTM in few-shot setting. Finn et al. [7] proposed Model-Agnostic Meta-learning (MAML) where a meta-learner is designed to adapt parameters for crossed tasks. TaskNorm [4] proposed a normalization technique for meta-learning tasks to fuse different normalization values for efficient optimization.
Our model is more related to metric-based methods, which aim to distinguish different classes of instances with distance metrics in their feature space. MatchNet [43] introduces cosine similarity to measure data distance and compute the similarity scores. ProtoNet [36] proposes the negative L2 distance to measure the difference between the class prototypes and the query vector. Deep-EMD [50] applies Earth Mover's Distance (EMD) as their distance metric. Recent methods transform the class prototypes to taskspecific prototypes with task-adaptive weights [19] or transformer [47]. The limitation of the existing FSL methods is that it only guarantees the good performance under the closed-set settings. The query set and the support set are in the same classes. FSOSR extends the FSL task with the unseen detection capability. We inject the additional background classes into conventional FSL methods and get a better detection capability.

Open-set recognition
Open-set recognition addresses the classification model with the detection capability of unseen samples [2,15,16,28,[38][39][40]48]. It aims to reject unseen samples using the open-set classifier and classify the seen samples. The model cannot see the training samples for open-set data, and can only use the training data from the close-set. Bendale et al. [2] train a close-set classification network first and then endow the trained network with open-set discrimination using Extreme Value Theory. Gopenmax [8] synthesizes fake open data during training. Neal et al. [28] propose to generate counterfactual images for the unknown class and train an (N+1)-class classifier to recognize the N known classes and 1 unknown class. Zhou et al. [58] also introduced the (N+1)-way classifier to solve the openset recognition problem, but they synthesize fake open data using manifold mixup [42] techniques, while we utilize image background. Recently, GAN-based methods gain popularity [1,8,16,28,29,39]. C2AE [29] and CGDL [39] use all the training samples to train an autoencoder and detect unseen samples by the large reconstruction error. Perera et al. [31] propose reconstructing the image along the channel dimension. The reconstruction of the unseen class samples is easy to fail which will lead to a lower classification probability for the unseen class. The normal OSR methods all require rich training data to train the unseen sample detector [2,28,29,31,39], which is hard to train in the FSL situation and getting poor performance.

Few-shot open-set recognition
FSOSR is recently proposed to address the open set recognition tasks in the few-shot inputs scenario [14,21]. PEELER [21] proposes the FSOSR task and solves the problem using the meta-learning framework and Mahalanobis distance to classify the known samples.
They introduce an open-set loss using a negative entropy loss to maximize the entropy for seen classes. SnaTCHer [14] uses the pre-train model with the transformer from FEAT [47] and needs to reuse the transformer to remap all the known and unknown query samples and a set of prototypes to a new space to calculate the distance. Our method is a metric-based FSOSR approach which is the same as PEELER and SnaTCHer, but we do not need the transformer module. PEELER [21] uses Mahalanobis distance. It needs to calculate the time-consuming covariance matrix. We follow MatchNet [43] to use cosine distance which is simple and effective.
FSOSR is different from the Generalized zero-shot learning [9,22] (GZSL). Although both do not have support samples for new classes, GZSL provides the semantic information for the target classes. FSOSR targets rejecting unseen new classes, the class information is unknown. As for GZSL, it is known information with class-level semantic description and target to classify target class samples to the correct classes.

PROBLEM SET-UP
FSOSR aims to design a machine learning algorithm that can reject unseen classes from only a few seen training examples and classify the seen classes samples correctly. 1 The typical setup for FSOSR follows episodic training and testing schema where batched tasks are sampled for training or testing. Specifically, an N-way K-shot FSOSR problem contains the support set and the query set for each episode.
For support set D = { , } =1 , where ∈ X and ∈ Y . The data from the support set X are from the classes seen with the label Y . The query set for FSOSR is different from FSL, D = D ∪ D , where D is the known query set and D is the unknown query set. The known query set are samples from the seen classes with the same label space  We introduce extra background classes to reserve space for unknown classes. Our background classifier uses background features as unknowns to optimize the classifiers. We use a ProCAM module to collect multiple normalized class activation maps to find the foreground mask. Query embedding can be a known query or an unknown query, which is encoded with the same backbone. We use cosine similarity to obtain the class score.
class. The unknown query is sample from the unseen class with There is no overlap for the seen labels and unseen Y ∩ Y = .
An algorithm fits model ( ) is defined as ( ) = W Φ( ), where Φ(·) : R → R is the embedding function and W ∈ R × is the Fully Connected(FC) layer to classify seen classes. We denote the classifier for class as w : W = [w 1 , · · · , w ].

METHOD
In this section, we introduce our framework for few-shot Openset Recognition. We first describe our extra background classifier method for clustering the unknown samples and separate the known classes in Section 4.1. Then we present our proposed background feature as unknowns schema to train the extra background classifiers in Section 4.2. The overview of the entire training pipeline is shown in Fig. 3.

Extra background classifier
The purpose of Few-shot Open Set Recognition (OSR) is to separate unknown samples from known samples. The known samples need to do close-set classification in further steps. The difficulty lies in that unknown samples are incorrectly classified as known samples if the classifier only contains known classes. For a CNN classifier with classes, it will divide the embedding space into parts. The class boundaries are determined by the discriminative information in the classifiers. Unknown samples projecting to the same embedding space will fall into known classes space and classify wrongly with such kinds of classifiers. Therefore, we propose to reserve space for unknown classes, which is simple and straightforward by adding extra class classifiers. We get the hint from a special class in segmentation problems, the background class, which may contain a lot of unknown information compared to the foreground known classes. We name our extra class as background class for easy understanding. Let W ∈ R × be the parameter matrix for the additional background classifier. It has the same number of feature channels as the known classes classifier W ∈ R × . is the number of known classes and is the number of background classes we pre-set to cluster possible unknown samples. The final FC of our network becomes W = [W , W ] ∈ R ×( + ) .
We employ the episodic training approach [36] to train our model. Each mini-batch of training will mimic an FSOSR task, the classifier weight for each known class can be seen as the average embedding for each class .
We introduced the background features as unknowns method to train our background class classifier using the background features of the support data. The details are provided in Section 4. 2 We use the cosine distance as our distance metric to measure the similarity between the query embedding Φ(X ) and the final FC of our network W.
The highest similarity of Φ( ) to the classifier weight w define the prediction of belong to . The loss can be expressed as:  whereˆ= argmax{sim⟨W , Φ( )⟩} + | | is the background class label with the maximum similarity the unknown query belongs to. The second term matches the unknown query to the reserved background classes space.

Background feature as unknowns
FSOSR lacks open-set data to train the model. The existing OSR method normally uses GAN to generate pseudo-samples for openset data, which is time-consuming and requires an extra branch of the generator network. We notice that the background of support data usually contains extra information and has a large difference from the foreground known class data. We can use background features as unknowns to train our extra background class and optimize the known classifier. Figure 4 shows the details of the classifier training. The training input is the support feature and the background feature of the support. The target label for the support feature is the known class label. We assign a background class label to the background feature. The backward propagation will only fine-tune the classifiers through the following loss.
The background feature can be extracted from the class activation map of different known classes. We proposed a ProCAM module that utilized multiple normalized class activation maps to mine class-related regions and get better background features.

ProCAM
The ProCAM module is developed to apply CAM on the original support features multiple times to figure out the previously uncovered known class regions. The maps found multiple times can be superimposed to get more complete background area. This section introduces details on how to generate multiple class activation maps. A class activation map for a specific category will highlight the discriminative regions for that category. For a given input image with labeled class , let ℎ 0 ( , ) represent the activation of the image in spatial location ( , ) in the last convolutional layer before average pooling. The CAM for class can be calculated through the following equation: where w is the classifier weight for class . We scale 0 ( ) to [0, 1] using the Min-Max normalization.
The first background feature map is getting through ℎ 1 = ℎ 0 × (1 − ( 0 ( ))). We regard the first background as the input of the CAM equation again, the CAM function will highlight more discriminative regions to the class .
By repeating this operation for times can give us different CAMs [ 0 , 1 , · · · , ( )]. We sum all these CAMs up and normalize them again to get the final ProCAM.
The final background feature map ℎ = ℎ 0 (1 − ), where ℎ 0 is the original feature map. We improve ProCAM using softmax normalization to emphasize the high response regions and squeeze the low response regions to get a better mask for the foreground area.

EXPERIMENT 5.1 Datasets
miniImageNet [43]: It is a popular benchmark for few-shot learning with 100 classes and 600 images in each class. [21] proposed to use this dataset for the FSOSR task. Following the conventional splitting [43] of meta-training(64), meta-validation (16), and metatesting (20) for FSL, the few-shot open-set problem samples training episodes from the 64 meta-training classes, and testing episodes from the 20 meta-testing classes.
tieredImageNet [33]: It is another popular FSL benchmark with more classes, which includes 608 classes from 34 super-classes. We follow [14] to split training, validation, and testing into 351, 97, and 160, respectively.
Caltech-UCSD Birds-200-2011 [44]: CUB is a fine-grained bird classification dataset with 200 categories and in total 11,788 images. Under the 5-way FSOSR setting, we randomly sampled 5 classes as the closes set and 5 classes as the open sets from 100 meta-training classes to form the training episodes. The validation episodes and testing episodes were sampled from the rest 50 meta-validation classes and 50 meta-testing classes.

Implementation Details
Following [14], we use the same network backbone Resnet12 for all datasets, where the last layer before the classifier contains 640 dimensions. We built our project using PyTorch [30]. The model was pre-trained with batch size of 128 for 500 epochs and optimized  ( ) ← Get CAM from ℎ ( ) and with eq. 5; 8: ( ( )) ← Get normalized CAM using eq. 6; loss3 ← Compute loss with L 3 ( ,ˆ), 16: loss4 ← Compute loss with L 4 ( ,ˆ), 17: loss=loss3+ loss4 ← get total loss, 18: {W ′ , W ′ } ← Update classifier with SGD 19: end while using SGD with momentum. The pre-training learning rate is adjusted using cosine annealing from 0.1 to 0 in a cycle of 100 epochs. Standard data augmentation with random crops, random color jitters, and random horizental flips are applied during pre-training. Manifold-mixup in conjunction with self-supervision [27] is also used to make backbone more robust. Our baseline is based on the first FSOSR paper [21] using meta-learning strategy to finetune the backbone episodically. We use cosine similarity as our distance metric and fine-tune the network in 20,000 iterations with an initial learning rate 0.0002. Ours ProCAM and softmax ProCAM are finetuning the extra background classifier with 0.002 initial learning rate for 20 epochs. The loss weight is set to 0.05 for miniImageNet, CUB, and 0.005 for tieredImageNet. We used the validation data to  select the best model and evaluated the test data for 600 episodes to minimize uncertainty. Evaluation Protocol. We follow [14,21] to evaluate the model in two metrics, the known class classification accuracy (acc) and the area under the ROC curve (AUROC) for the unknown class detection. The known class accuracy takes the known query ground truth label and checks the model corrected classification ratio.  Table 3: Comparison with the State-of-the-art methods. Average closed-set classification accuracies (%) and average unknown detection AUROCs (%) over 600 episodes for MiniImageNet and TieredImageNet. PEELER* is quoted from the paper, which has a ResNet-18 backbone.  Table 4: Comparison with the State-of-the-art methods. Average closed-set classification accuracies (%) and average unknown detection AUROCs (%) over 600 episodes for CUB.
is plotted based on the different thresholds for the true positive rate versus the false positive rate. The higher AUROC indicates the better unseen detection capability.

Analysis
We evaluate the effectiveness of different initialization methods for the background class in Table. 2. For each initialization method, we compare the different number of fine-tune epochs for the background class classifier learning. The random initialization method uses the Kaiming uniform initialization [10]. The 'avg' method initializes the background classifier weight with the average background feature maps. The 'global' method uses a global FC for background class across all the training and testing episodes. The results show that random initialization for every episode and fine-tuning with 20 epochs have a better result compared to the other methods. Fig. 5 shows the accuracy and AUROC change w.r.t. the number of repeating ProCAMs and the number of additional background classes. There is an increasing trend for accuracy and AUROC when the number of repetitions increases and reaches a maximum at = 4. Increasing the number of extra background classes does not affect the accuracy too much, but the AUROC changes a lot. The highest AUROC for miniImageNet 5-shot is adding 2 extra background classes. known class prototypes class proto after fine-tune known class support known query unknown query background features Figure 6: T-SNE [26] visualization of data embeddings and classifiers before (left) and after (right) fine-tune with background features as unknowns. Crosses with different colors represent query data points from different known classes. The pink plus represents unknown query. The blue squares and triangles indicate the classifier prototypes before and after fine-tuning, respectively. Our module reserve the space for the background classifier as shown in the green area.
We also plot the t-SNE figure [26] to see the update of the feature space for the background class. Fig. 6 shows that our proposed background classifier helps reserve space for unknown samples and provides a direction for the unknown class to cluster to the extra background class centers.
We provide a visualization of our ProCAM and softmax ProCAM, and compared them with the original CAM in Fig. 7. Our methods generate more accurate foreground masks, which cover all the important features in the known classes. The background feature based on our mask contains less discriminative information for seen class and can provide good background data to train our extra background classes. The background feature data is also helpful for pushing the boundary of known classes. Softmax ProCAM is similar to the ProCAM results, the background and foreground boundaries are more clear than ProCAM.
Ablation study. We perform our ablation study on the 5-way 5-shot setting for tieredImageNet to analyze the effectiveness of different components in 1. The base method is the testing result   [57]. Ours can generate more accurate foreground masks and get better background features that contain less information from the known classes. Our Softmax ProCAM generates better background features through the well-defined boundaries (The background regions heatmap is darker compared to the others).
for the pre-train model under FSOSR setting directly. The method 'meta_cos' employs the meta-learning and cosine distance and gets a better result than the pre-training model. We apply the vanilla class activation map to the previous 'meta_cos' method and we can get higher accuracy with better AUROC for this 'meta_cam' model. The previous CAM method only uses the background feature to fine-tune the background class classifiers, we try to fine-tune all the FCs using both the support feature and the background feature in 'meta_cam_ft'. The result provides around 0.6% of the increase in the accuracy, which indicates that the background features can help push the boundaries of the known classes and train better background class classifiers. We optimize the fine-tune of all FCs with better background feature maps. The results show an increasing trend in known class accuracy and unknown class AUROC.

Comparison with the State-of-the-Art Methods
Finally, we conduct comparison of our performance with the stateof-the-art results on three benchmarks: miniImageNet, tieredIm-ageNet, and CUB. We choose the state-of-the-art FSL methods ProtoNet [36], FEAT [47], and ASY [3] as the first group of models. In the second group, we select two OSR methods (NN [15], Open-Max [2]). The third group is our baseline with mete-learning and cosine classifier and the current existing FSOSR methods PEELER and SnaTCHer [14,21]. Because the existing FSL and OSR methods are not designed for FSOSR tasks, we follow SnaTCHer [14] to slightly modify them for comparison. We show the detailed numbers for miniImageNet and tiered-ImageNet in Table 3. The comparison for CUB is in Table 4. Our model has the highest AUROC of all the methods. The accuracy of the known classes is also competitive among all the datasets. In particular, our AUROC outperforms the state-of-the-art result by 3.14% on miniImageNet 5-way 1-shot. Our accuracy increased 1.64% for the same dataset 5-shot. For tieredImagenet, the 1-shot accuracy is slightly lower maybe due to the lack of support background data, but the AUROC still improved around 1.27%. The accuracy and AUROC on CUB outperform the state-of-the-art method by 10.56% and 11.5%, respectively.

CONCLUSION
In this paper, we solve the few-shot open-set recognition problems from two aspects. We first introduced extra background class classifiers to reserve space for unknown samples, which use cosine distance to determine the space for each class. Then, we propose a learning paradigm to utilize the background features as unknowns to train the extra background class classifiers and fine-tune the known classes' FCs. To generate a better mask for the foreground objects and get a better background feature, we propose a ProCAM module to utilize multiple class activation maps. We optimize our ProCAM module using softmax to separate the foreground part and generate clear boundaries of the background. Experiments on three datasets show that our method significantly outperforms the baselines and the state-of-the-art approaches.