Open Long-Tailed Recognition in a Dynamic World

Real world data often exhibits a long-tailed and open-ended (with unseen classes) distribution. A practical recognition system must balance between majority (head) and minority (tail) classes, generalize across the distribution, and acknowledge novelty upon the instances of unseen classes (open classes). We define Open Long-Tailed Recognition++ (OLTR++) as learning from such naturally distributed data and optimizing for the classification accuracy over a balanced test set which includes both known and open classes. OLTR++ handles imbalanced classification, few-shot learning, open-set recognition, and active learning in one integrated algorithm, whereas existing classification approaches often focus only on one or two aspects and deliver poorly over the entire spectrum. The key challenges are: 1) how to share visual knowledge between head and tail classes, 2) how to reduce confusion between tail and open classes, and 3) how to actively explore open classes with learned knowledge. Our algorithm, OLTR++, maps images to a feature space such that visual concepts can relate to each other through a memory association mechanism and a learned metric (dynamic meta-embedding) that both respects the closed world classification of seen classes and acknowledges the novelty of open classes. Additionally, we propose an active learning scheme based on visual memory, which learns to recognize open classes in a data-efficient manner for future expansions. On three large-scale open long-tailed datasets we curated from ImageNet (object-centric), Places (scene-centric), and MS1M (face-centric) data, as well as three standard benchmarks (CIFAR-10-LT, CIFAR-100-LT, and iNaturalist-18), our approach, as a unified framework, consistently demonstrates competitive performance. Notably, our approach also shows strong potential for the active exploration of open classes and the fairness analysis of minority groups.


INTRODUCTION
O UR visual world is inherently long-tailed and open-ended [1], [2], with a few common visual categories (i.e., head classes) and many more relatively rare categories (i.e., tail classes). At the same time, new visual concepts constantly emerge as we navigate in an open world (i.e., open classes).
Although natural data distributions contain head, tail, and open classes, existing classification approaches focus mostly on either the head [3], [4] or the tail [5], [6], and often in a closed setting [7], [8] (Fig. 2). We thus formally study Open Long-Tailed Recognition++ (OLTR++) arising from natural data settings. A practical system should be able to work for a few head and many tail categories, to generalize the concept of a single category from only a few known instances, as well as to acknowledge and explore novelty upon an instance of an unseen or open category. We define OLTR++ as learning from long-tail and open-end distributed data and evaluating the classification accuracy over a balanced test set which includes head, tail, and open classes in a continuous spectrum (Fig. 1).
The key challenges for OLTR++ are tail recognition robustness and open-set sensitivity. As the number of training instances drops from thousands in the head class to a few in the tail class, we should prevent the performance from dropping drastically.
Meanwhile, for open classes, the recognition performance relies on the sensitivity to distinguish unknown samples from known classes, as well as to select informative samples for data-efficient active exploration and future model updates.

Dynamic World
Imbalanced Classification

Head Classes Tail Classes Open Classes
Open Set Recognition + Active Learning In our algorithm (OLTR++), there is a learned metric that respects both the closed-world classification and acknowledge the novelty of the open world. This metric maps images to an embedding space (dynamic meta-embedding), where visual concepts can relate to each other to improve both the tail recognition robustness and open-set sensitivity.
Specifically, our dynamic meta-embedding is a combination of two components: direct feature and induced feature. 1) Direct feature is a standard embedding that gets updated from the training data by stochastic gradient descent over the classification loss. It is usually less generalized in tail classes, compared to head classes, because of the lack of sufficient supervision. 2) Memory feature, instead, is an induced feature through a visual memory association mechanism, inspired by meta-learning methods [5], arXiv:2208.08349v1 [cs.CV] 17 Aug 2022 Fig. 2: Comparison between our proposed OLTR++ and related existing tasks. Our task, in a more realistic setting, is a combination of the three existing tasks (imbalanced classification, few-shot learning, and open-set recognition) in addition to active exploration, which constitutes a dynamic learning loop that can facilitate robust deployment of vision systems. [9], [10], that augments the direct feature of each image for better distinguishability. The association is learned to retrieve a summary of memory activations from the direct features of each input and then combined with the original direct features to be the metaembeddings. This memory feature augmentation is particularly effective on tail classes for the lack of supervision to provide generalized features . We address open-class sensitivity by dynamically calibrating the meta-embedding with respect to the visual memory. In other words, the embedding is scaled inversely by its distance to the nearest class centroid (i.e., class memory): the farther away it is from the memory, the more likely it is an open set instance. The distance between the embedding and the nearest centroid is then transformed into "sample informativeness" using an energybased model [11], which is further employed to select informative samples for active learning.
We also adopt modulated attention [12] to encourage the head and tail classes to use different sets of spatial features. As our meta-embedding relates head and tail classes, our modulated attention maintains discrimination between them.
We make the following major contributions. 1) We formally define the OLTR++ task, which learns from long-tail and openend distributed data and optimizes for the overall accuracy over a balanced test set. It provides a comprehensive and unbiased evaluation of visual recognition algorithms in practical settings. 2) We propose an integrated OLTR++ algorithm with dynamic meta-embedding. It handles tail recognition robustness by relating visual concepts among head and tail embeddings, and it handles open-class sensitivity by dynamically calibrating the embedding with respect to the visual memory. 3) We further incorporate an energy-based model into our dynamic meta-embedding for dataefficient active learning (with only sparse human annotations), which is well suited for the constantly-changing visual world. 4) We curated three large OLTR++ datasets according to a longtailed distribution from existing representative datasets: objectcentric ImageNet [3], scene-centric Places [13], and face-centric MS1M datasets [14]. We also set up benchmarks for proper OLTR performance evaluation. 5) Our extensive experimentation on these OLTR++ datasets (as well as standardized benchmarks such as CIFAR-LT 100/10 and iNaturalist-18 [15]) demonstrates that our method consistently outperforms the state-of-the-art methods.
The aim of this work is to advocate a new learning paradigm that can perceive and update in a dynamic world, i.e. simultaneously recognizing real-world long-tailed data while actively exploring novel data with human in the loop. It is a crucial step towards embodied intelligence as well as the robust deployment of vision systems. Besides exhibiting competitive performance on long-tailed recognition, our approach also demonstrates compelling results on open class detection and active exploration with a unified framework centered around visual memory.
Our code, datasets, and models are publicly available at https:// liuziwei7.github.io/projects/LongTail.html. Our work fills the void in practical benchmarks for imbalanced classification, few-shot learning, open-set recognition, and active learning, enabling future research that is directly transferable to real-world applications.

RELATED WORKS
While OLTR++ has not been defined in the previous literature, there are four closely related tasks which are often studied in isolation: imbalanced classification, few-shot learning, openset recognition, and active learning. Fig. 2 summarizes their connections and key differences. Imbalanced & Long-Tailed Recognition. Imbalanced classification is an extensively studied area [24], [25], [26], [27], [28], [29], [30], [31], [32]. While classical methods include under-sampling head classes, over-sampling tail classes, and data instance reweighting, some recent methods apply metric learning [33], [34], hard negative mining [17], [35], and meta learning [7], [36]. For example, lifted structure loss [34] introduces margins between many training instances. The range loss [37] enforces data in the same class to be close and those in different classes to be far apart. Focal loss [17] induces an online version of hard negative mining. And metaModelNet [7] learns a meta regression net from head classes and uses it to construct the classifier for tail classes.
As a step forward from imbalanced classification, longtailed recognition (where the training datasets are more imbalanced) [18], [19], [20], [21], [38], [39], [40], [41], [42] Method Formulation Venue CIFAR-10 CIFAR-100 ImageNet-LT Places-LT  has attracted extensive research interests recently. In Table 1, we provide a systematic overview of representative long-tailed recognition works (including those published after ours). In our approach, we have a dynamic meta-embedding that combines the strengths of both metric learning and meta learning. On the one hand, our direct feature is updated to ensure centroids of different classes are separated from each other; On the other hand, our memory feature is generated on-the-fly in a meta learning fashion to effectively transfer knowledge from head classes to tail classes. Few-Shot Learning. Few-shot learning is often addressed by meta-learning techniques [43], [44], [45], [46], [47], [48]. For example, matching Network [5] learns a transferable feature matching metric to go beyond given classes. And Prototypical Network [49] maintains a set of separable class templates. Additionally, feature hallucination [50] and augmentation [51] have also shown effectiveness. Since these methods focus on fast learning from novel and unseen classes, they often suffer a "catastrophic forgetting" for training classes . Few-shot learning without forgetting [52] and incremental few-shot learning [53] attempt to remedy this issue by leveraging the duality between features and classifiers' weights [54], [55]. However, these methods rely on balanced training sets, while OLTR++ learns from naturally long-tailed training sets instead.
Our approach is closely related to meta learning with associative memory [5], [9], [10], [56], [57], [58]. Compared to prior arts, our memory feature has two advantages: 1) it transfers knowledge to both head and tail classes adaptively via a learned concept selector; 2) it is fully integrated into the network without episodic training, thus suitable for large-scale applications. Open-Set Recognition. Open-set recognition [59], [60], or outof-distribution detection [61], [62], [63], aims to re-calibrate the sample confidence in the presence of open classes. One of the representative techniques is OpenMax [60], which fits a Weibull distribution to model logits. However, when there are both open and tail classes, distribution-fitting based methods often confuse the two because of the less generalized features of tail classes. Instead of calibrating the output logits, our OLTR++ approach incorporates a confidence estimation mechanism into feature learning and dynamically re-scale the meta-embedding w.r.t. the learned visual memory, such that samples from known classes are expected to be closer to the memory compared to novel samples. Open-World Recognition. Open-world recognition [64], [65], [66] is a closely related field whose goal is to distinguish "unknown unknown classes" from "known known classes". [64] also considers a dynamic setting where unknown classes are continuously added and detected, and examines the influence of unknown classes on the accuracy of known classes. [66] further incrementally learns the new classes. Once they are detected as unknown and an oracle provides labels for the objects of interest among all the unknowns. Here we advocate this dynamic-world endeavor: instead of just detecting the unknown classes, we aim to recognize the semantic label of the unknown classes. Zero-Shot Learning. Zero-shot learning (ZSL) [67], [68], [69] is also a promising direction for recognizing novel classes. ZSL aims to learn the association between base and novel class features with the aid of certain shared semantic knowledge (e.g. attributes, word2vec), which is not directly applicable here. In comparison, our active exploration is more focused on the annotation-efficiency of recognizing novel classes, i.e. using less human annotations to achieve acceptable accuracies on novel classes. Active Learning. Active learning aims to explore unlabeled data with an oracle annotator that provides ground truth labels to a few selected samples. The central issue here is the exploration efficiency, i.e. obtaining higher performance with less oracle queries. The representative works can be roughly categorized into two realms: generation-based methods [70], [71] and selectionbased methods [72], [73].
However, existing active learning methods mainly work in closed-world setting, where they focus on selecting informative samples for the known categories to improve the performance. Here we study a realistic yet more challenging problem, selecting informative samples from a mixture of known and unknown categories so as to recognize both of them, which we coined as "active exploration". Their key differences are listed in Table 2. Our active exploration considers both informativeness and openness during sample selection. Incremental Learning. Incremental learning aims to continually learn new tasks (e.g. novel classes) without catastrophic forgetting. Extensive research has been performed from different perspectives: neural architectures [74], [75], replay/rehearsal mechanisms [76], [77], parameter regularization [78], [79] and learning techniques [80], [81]. Here our dynamic learning loop   aims to achieve continual recognition of novel classes in a dataefficient manner.
Fairness Learning/Analysis. The open long-tailed recognition proposed in our work also has an intrinsic relationship to fairness analysis [82], [83], [84], [85], [86]. Their key differences are listed in Table 3. On the problem setting side, both open longtail recognition and fairness analysis aim to tackle the imbalance existed in real-world data. Open long-tailed recognition focuses on the longtail-ness in both known and unknown categories while fairness analysis deals with the bias in sensitive attributes such as male/female and white/black. On the methodology side, both open long-tailed recognition and fairness analysis aim to learn transferable representations. Open long-tailed recognition optimizes for the overall accuracy of all categories while fairness analysis optimizes for several attribute-wise criteria. The preliminary results in Table 6 demonstrates that our proposed dynamic meta-embedding is also a promising solution to fairness analysis.

OUR OLTR++ MODEL
We propose to map an image to a feature space such that visual concepts can relate to each other based on a learned metric that respects the closed-world classification while acknowledging the novelty of the open and dynamic world. Our model has three major modules (Fig. 3): dynamic meta-embedding, modulated attention, and active exploration. The first relates and transfers knowledge between head and tail classes, and the last two maintain discrimination between them with human-in-the-loop.

Intuitive Explanation of Our Approach
In this section, we give an intuitive explanation of our approach that tackles the problem open long-tailed recognition. From the Firstly, in our approach, we obtain a visual memory by aggregating the knowledge from both head and tail classes. Then the visual concepts stored in the memory are infused back as associated "fast feature" to enhance the original "slow feature". In other words, we use induced knowledge (i.e. "fast feature") to assist the direct observation (i.e. "slow feature"). We further learn a concept selector to control the amount and type of infused "fast feature". Since head classes already have abundant direct observation, only a small amount of "fast feature" is needed. On the contrary, tail classes suffer from scarce observation, the associated visual concepts in "fast feature" are more beneficial to tail classes than to head classes. Finally, we calibrate the confidence of open classes by calculating their reachabilities (i.e. feature space distances) to the obtained visual memory (i.e., class centroids). All together, we provide a comprehensive treatment to the full spectrum of head, tail and open classes, improving the performance on all categories.

Dynamic Meta-Embedding
Dynamic meta-embedding is a combination of a direct image feature and an associated memory feature, where the feature norm indicates the familiarity to known classes.
Consider a convolutional neural network (CNN) with a softmax output layer for classification. The second-to-the-last The discriminative centroids constitute our visual memory, which are obtained with two iterative steps, neighborhood sampling and affinity propagation.
layer is the feature and the last layer is the linear classifier (cf. φ(·) in Fig. 3). The feature and the classifier are jointly trained in an end-to-end fashion. Let v direct denote the direct feature extracted from an input image. The final classification accuracy largely depends on the quality of this direct feature. While a feed-forward CNN classifier works well with big training data [3], [87], it lacks sufficient supervised updates from small data in tail classes. We propose to enrich direct feature v direct with visual concepts in a memory module through a memory feature v memory , which is derived from a memory bank that captures visual concepts of each training classes. This mechanism is similar to the memory components in meta learning [46], [58]. We denote the resulting feature meta embedding v meta . Learning Visual Memory M . To construct the memory bank, we follow [88], [89] on class structure analysis and adopt discriminative centroids as the basic building block. Let M denote the visual memory of all the training data, where K is the number of training classes. Compared to alternatives [49], [90], this memory is appealing for our OLTR++ task: it is almost effortlessly and jointly learned alongside the direct features {v direct n }, and it considers both intra-class compactness and interclass discriminativeness.
More concretely, as illustrated in Fig. 5, we compute class centroids in two steps. 1) Neighborhood Sampling: We sample both intra-class and inter-class examples to compose a mini-batch during training. These examples are grouped by their class labels and the centroid c i of each group is updated by the direct feature of this mini-batch, which can be formulated as: where 1(·) is the indicator function, and B is the batch size.
2) Affinity Propagation: We alternatively update the direct feature v direct and the centroids to minimize the distance between each direct feature and the corresponding class centroids and maximize the distance to other centroids. Note that the "repelling edges" in Fig. 5 are calculated through a large margin loss L LM as described in Eqn. 13.
At the end of the training, we obtain a visual memory module M containing important visual concepts within the dataset.
Composing Memory Feature v memory . For an input image, v memory enhances its direct feature when training data are limited (as in the tail class). The memory feature relates class centroids in the memory to transfer knowledge to tail classes: where o ∈ R K is the coefficients hallucinated from the direct feature. We use a lightweight neural network to obtain the coefficients from the direct feature, o = T hal (v direct ).
Obtaining Dynamic Meta-Embedding. v meta combines the direct feature and the memory feature, and is fed to the classifier for the final class prediction (Fig. 6): where ⊗ denotes element-wise multiplication and e is a concept selector. γ > 0 is a seemingly redundant scalar for the closedworld classification tasks. However, in the OLTR++ setting, it serves an important role in differentiating the examples of the training classes from those of the open-set. γ measures the reachability [91] of an input's direct feature v direct to the memory M -the minimum distance between the direct feature and the discriminative centroids: When γ is small, the input likely belongs to a training class from which the centroids are derived, and a large reachability weight 1/γ is assigned to the resulting meta-embedding v meta . Otherwise, the embedding is scaled down to an almost all-zero vector at the extreme, which is useful to encode open classes.  Fig. 7: Modulated attention is spatial attention applied on selfattention maps ("attention on attention"). It encourages different classes to use different contexts, which helps maintain the discrimination between head and tail classes.
As the direct feature is often good enough for the data-rich head classes, whereas the memory feature is more important for the data-poor tail classes. We design a concept selector, e, to adaptively select them in a soft manner. We learn a lightweight network T sel (·) with a tanh(·) activation function on v direct :

Modulated Attention
Since discriminative cues of head and tail classes tend to distribute at different locations in the image, we find it beneficial to enhance direct feature, v direct , with spatial attention to separate head and tail classes. Specifically, we propose modulated attention to encourage samples of different classes to use different contexts. Firstly, we compute a self-attention map SA(f ) from the input feature map by self-correlation [12]. It is used as contextual information and added back (through skip connections) to the original feature map. The modulated attention M A(f ) is then designed as conditional spatial attention applied to the selfattention map: M A(f )⊗SA(f ), which allows examples to select different spatial contexts (Fig. 7). The final attention feature map becomes: where f is a feature map in CNN, SA(·) is the self-attention operation, and M A(·) is a conditional attention function [92] with a softmax normalization.
Sec. 4.1 shows empirically that our attention design achieves superior performance than the common practice of applying spatial attention to the input feature map. This modulated attention ( Fig. 7 (b)) could be plugged into any feature layer of a CNN. Here, we modify the last feature map only.

Active Exploration of Open Classes
In the dynamic world, the model should not halt after training. We assume a continuous training, inference, annotation, and model update loop as our model actively explores the visual world Compute the ratio between the first two nearest distances: Compute the free energy function of v direct : E n ← −T act · log K i e Uopen·U inf o /Tact . Select high-energy samples for further human annotations. Update the classifier φ(·) using newly added data. end for over time. Every time our model encounters certain sample of open classes, our model will determine whether this sample is informative enough for further human annotation. After obtaining these human annotations in an efficient manner, our model will be continually updated according to the newly added data.
The active exploration step has three major components: 1) active sample selection based on two different types of uncertainty, 2) human-in-the-loop annotation, and 3) model update using active data annotations, all three of which constitute a dynamic recognition loop. The detailed algorithmic pipeline of our active exploration is listed in Alg. 1.

Two Types of Uncertainty in Active Exploration
Unlike the standard active learning setting that work in closedworld setting, there actually exist two types of uncertainty here in active exploration: uncertainty in openness and uncertainty in informativeness. Existing active learning algorithms are not directly applicable here since their uncertainty estimation mechanism only considers the informativeness among known classes, which is not suitable for modeling the openness between known classes and unknown classes.
In the following, we elaborate the modeling the two types of uncertainty in the context of active exploration: 1) Uncertainty in Openness: We measure the openness U open of a new sample using the distance between its embedding and the nearest centroid, which can be formulated as: where v direct is the direct feature of the new sample and c i is the centroid of the i-th class. 2) Uncertainty in Informativeness: Intuitively, the most informative samples would be those that lie on the decision boundaries between different classes. We first sort the distances between the embedding of the new sample to all existing class centroids in ascending order: d sorted n . Then the informativeness of a new sample is defined as the ratio between the first two nearest distances: These two types of uncertainty regarding new sample and class centroids are further illustrated in Fig. 8.

Active Sample Selection
At each time step when new data are encountered, the selection of samples for active annotations is based on the combination of both openness and informativeness uncertainty estimation using the energy-based model [11]. Similar to [63], we can express the free energy function E(·) of new sample v direct as follows: where T act is the temperature for controlling the trade-off between precision and recall in active selection.

Human-in-the-Loop Annotation
After selecting these "open" yet "informative" samples, we obtain the semantic labels of these samples by querying human annotators. In the real-world applications [93], [94], it can be executed through online crowdsourcing platform with quality control. Since different human annotators have different preferences for naming unknown categories, it is crucial to maintain consistency between different samples of the same unknown category.

Model Update
Assume that we have an existing classifier φ t (·) at time step t with weight vectors {w i } K i=1 for K known classes. And during time step t to time step t + 1, we will encounter Z unknown classes. Then at time step t + 1, our new classifier φ t+1 (·) will be a concatenation of both known class weights and unknown class weights: Both weights will be updated by the obtained active human annotations.
Note that model update is not our main contribution; there are several feasible instantiations. Since we adopt cosine classifer described in Sec. 3.5, the weight of classifier and the embedding of sample can be transformed interchangeably [52], [55]. Though more sophisticated methods (e.g. learning another network to generate the classifier weights for unknown classes) can be applied, here the classifier weights for unknown classes are simply hallucinated through a weighted average of the meta-embeddings from the actively selected samples for that class, where the weight is determined by E(v direct ). This classifier hallucination approach is extremely suitable for off-the-shelf deployment.

Dynamic Recognition Loop
To accommodate for the dynamic nature of the visual world, this procedure of active sample selection, human-in-the-loop annotation, and model update repeats each time new batch of open data are encountered. Our framework maximizes learning and recognition efficiency by taking the best from both humans and machines within a synergistic collaboration, taking care of both the long-tailed and open-ended distribution existing in the natural world.
In our implementation, the feature extractor is fixed for fast adaptation, while the visual memory is updated to accommodate for the continual stream of unknown classes during the dynamic learning loop. We have further clarified in the revised paper.

Learning
Cosine Classifier. We adopt the cosine classifier [52], [55] to produce the final classification results. Specifically, we normalize the meta-embeddings {v meta n }, where n stands for the n-th input as well as the weight vectors The normalization strategy for the meta-embedding is a nonlinear squashing function [95] which ensures that vectors of small magnitude are shrunk to almost zeros while vectors of big magnitude are normalized to the length slightly below 1. This function helps amplify the effect of the reachability γ (cf. Eq. (3)).
Loss Function. Since all our modules are differentiable, our model can be trained end-to-end by alternatively updating the centroids {c i } K i=1 and the dynamic meta-embedding v meta n . The final loss function L is a combination of the cross-entropy classification loss L CE and the large-margin loss between the embeddings and the centroids L LM : where λ is set to 0.1 in our experiments via observing the accuracy curve on validation set. Specifically, L CE is the cross-entropy loss between dynamic meta-embedding v meta n and the ground truth category label y n : where φ(·) is the cosine classifier described in Eqn. 6 in the main paper. Next we introduce the large margin loss L LM between the embedding v meta n and the centroids {c i } K i=1 : where m is the margin and we set it as 5.0 in our experiments. With this formulation, we minimize the distance between each embedding and the centroid of its group and meanwhile maximize the distance between the embedding and the centroids it does not belong to.

Datasets.
We curated three open long-tailed benchmarks, ImageNet-LT (object-centric), Places-LT (scene-centric), and MS1M-LT (face-centric), respectively.   ArcFace dataset [14], [97], we sampled images for each identity with a probability proportional to the image numbers of each identity. It has 887.5K images and 74.5K identities, with a long-tailed distribution. To inspect the generalization ability of our approach, the performance is evaluated on the MegaFace benchmark [98], which has no identity overlap with MS1M-ArcFace. Network Architectures. Following [50], [51], [52], we employed a ResNet-10 [16] trained from scratch as our backbone network for ImageNet-LT. To make a fair comparison with [7], the pre-trained ResNet-152 [16] was used as the backbone network for Places-LT. For MS1M-LT, we used the popular pre-trained ResNet-50 [16] as the backbone network. To compare with the recent works, we also adopted the two-headed ResNet-50 backbone following [23]. Evaluation Metrics. The evaluation is on the performance of each method under both the closed-set (test set contains no unknown classes) and open-set (test set contains unknown classes) settings. Under each setting, besides the overall top-1 classification accuracy [52] over all classes, we also calculated the accuracy of three disjoint subsets: many-shot classes (classes each with over training 100 samples), medium-shot classes (classes each with 20∼100 training samples) and few-shot classes (classes under 20 training samples). For the open-set setting, the F-measure was also reported for a balanced treatment of precision and recall following [60]. For determining open classes, the sof tmax probability threshold was initially set as 0.1, while a more detailed analysis is provided in Sec. 4.6. Comparison Methods. We chose for comparison state-of-theart methods from different fields dealing with the open longtailed data, including: (1) metric learning: Lifted Loss [34], (2) hard negative mining: Focal Loss [17], (3) feature regularization: Range Loss [37], (4) few-shot learning: FSLwF [52], (5) longtailed modeling: MetaModelNet [7], and (6) open-set detection: Open Max [60]. We applied these methods on the same backbone networks as ours for a fair comparison. We also enabled them with class-aware mini-batch sampling [100] for effective learning. Since Model Regression [101] and MetaModelNet [7] are the most related to our work, we recorded our results along with the numbers reported in the original papers. We also included the recent advances (e.g. CB Focal [18], LDAM [20], Decoupling [21], BBN [22], and RIDE [23]) in long-tailed recognition for a comprehensive evaluation.

Ablation Study
Effectiveness of the Dynamic Meta-Embedding. From Fig. 9 (b), we observe that the combination of the memory feature and concept selector led to large improvements on all three shots, because the obtained memory feature transferred useful visual concepts among classes. Another observation is that the confidence calibrator is the most effective on few-shot classes. And the reachability estimation inside the confidence calibrator helped distinguish tail classes from open classes. Effectiveness of the Modulated Attention. We observe from Fig. 9 (a) that, compared to medium-shot classes, the modulated attention contributed more to the discrimination between manyshot and few-shot classes in the experiments. Fig. 9 (c) further validates that the modulated attention is more effective than directly applying spatial attention on feature maps. It implies that adaptive contexts selection is easier to learn than the conventional feature selection. Effectiveness of the Reachability Calibration. To further demonstrate the merit of reachability calibration for open-world setting, we conducted additional experiments following the standard settings in [62], [96] (CIFAR100 + TinyImageNet(resized)). Our approach shows favorable performance over standard open-set methods [62], [96], as listed in Table 5.

OLTR++ Benchmarking Results
In this section, we extensively evaluate the performance of various representative methods on our OLTR++ benchmarks. ImageNet-LT.   both Lifted Loss [34] and Focal Loss [17] greatly boosted the performance of few-shot classes by enforcing feature regularization. However, they also sacrificed the performance on manyshot classes since there are no built-in mechanism of adaptively handling samples of different shots. Secondly, OpenMax [60] improved the results under the open-set setting. However, the accuracy degraded when it was evaluated with F-measure, which considers both precision and recall in open-set. This proves that when the open classes are compounded with the tail classes, it becomes challenging to perform the distribution fitting that [60] requires. Lastly, although the few-shot learning without forgetting approach [52] retained the many-shot class accuracy, it had difficulty dealing with the imbalanced base classes which are lacked in the current few-shot paradigm. Overall, as demonstrated in Fig. 10, our approach provides a comprehensive treatment to all the many/medium/few-shot classes as well as the open classes with substantial improvements on all aspects. Table 4 (b). With a much stronger baseline (i.e. ImageNet pre-trained ResNet-152), our approach still consistently outperformed other alternatives under both the closedset and open-set settings. The advantage is even more profound under F-measure.

MS1M-LT.
We trained on the MS1M-LT dataset and report results on the MegaFace identification track, which is a standard benchmark in the face recognition field. Since the face identities in the training set and the test set are disjoint, we adopted an indirect way to partition the testing set into subsets of different shots. We approximated the pseudo shots of each test sample by counting the number of training samples that are similar to it by at least a threshold (feature similarity greater than 0.7). Apart from many-shot, few-shot, one-shot subsets, we also obtained a zeroshot subset, for which we could not find any sufficiently similar samples in the training set. It can be observed that our approach has the most advantage on one-shot identities (3.0% gains) and zero-shot identities (1.8% gains) as shown in Table 6.

SUN-LT.
To directly compare with [101] and [7], we also tested out method on the SUN-LT benchmark they provided. The final results are listed in Table 8. Instead of learning a series of classifier transformations, our approach transfered visual knowledge among features and achieved an 1.4% improvement over the prior best.    Note that our approach also incured much less computational cost since MetaModelNet [7] requires a recursive training procedure.

Performance on Fairness Analysis
Results. On the sensitive attribute performance on MS1M-LT, the last six columns in Table 6 show that our approach achieved across-board improvements on both gender sub-groups and ethnicity sub-groups, which has an encouraging implication for effective fairness learning.
Indications. The goal of evaluating the performance on different gender sub-groups and ethnicity sub-groups here is to draw connection between long-tailed recognition and the larger community of bias and fairness in artificial intelligence, which could possibly motivate more future research at the intersection of these two.
To provide a comprehensive comparison on this interdisciplinary sub-field, we further implement and include two representative methods in fairness learning (i.e. fair feature [83] and latent debiasing [99]) into the evaluation.   [72], core-set-based method CoreSet [102] and adversariallearning-based method VAAL [73]. As shown in Fig. 12 (a), our OLTR++ validates its advantage over different percentages of selected open samples for active annotation and demonstrates dynamic-world learning efficiency. Analysis. To further understand the active exploration ability of OLTR++, we visualized the samples of selected and not selected images by our active exploration approach in Fig. 12 (b). We can observe that OLTR++ tends to select canonical images with representative parts and appearance for unseen classes.
Sorted Class ID w.r.t Occurrences Number of Images Per Class Absolute F1 Score Gains

Effectiveness of Dynamic Recognition Loop
Setting. The experiments were performed in a similar setting to active exploration, except that the open/exploration set was divided into two subsets to mimic the two stages in incremental/continual learning scenario. Specifically, both Stage 1 and To mimic the real-world applications, the accuracies of "known/unknown" is calculated by classifying test samples to a combination of known and unknown labels. Results. We compared our results with several representative methods in incremental/continual learning, including Learning without Forgetting (LwF) [80], and Adversarial Continual Learning (ACL) [81]. LwF [80] is a seminal work in the field of incremental learning, which serves as a strong and musthave baseline by employing a soft multi-task learning paradigm. ACL [81] is a recent adversarial-learning-based method of stateof-the-art performance without using external data or models. It is also equipped with open-sourced code and evaluation scripts for fair comparisons. As listed in Table 9, our OLTR++ maintains the learning and recognition effectiveness during the dynamic loop.

Influence of Tailness and Openness
Influence of Dataset Longtail-ness. The longtail-ness of the dataset (e.g. the degree of imbalance of the class distribution) can have a negative impact on the model performance. For faster investigating, the weights of the backbone network were frozen during training. From Fig. 11 (a), we observe that as the dataset became more imbalanced (i.e. power value α decreases), our approach only underwent a moderate performance drop. In other words, dynamic meta-embedding enables effective knowledge transfer among data-abundant and data-scarce classes.  Fig. 13: Examples of the top-3 infused visual concepts from memory feature. Except for the bottom right failure case (marked in red), all the other three input images were misclassified by the plain model and correctly classified by our model. For example, to classify the top left image which belongs to a tail class 'cock', our approach learned to transfer visual concepts that represents "bird head", "round shape" and "dotted texture" respectively.     [27].
Influence of Open-Set Probabilistic Threshold. The performance change w.r.t. the open-set probability threshold is demonstrated in Fig. 11 (b). Compared to the plain model [16] and range loss [37], the performance of our approach changed steadily as the open-set threshold rose. The reachability estimator in our framework helped calibrate the sample confidence, thus enhanced robustness to open classes. Influence of the Number of Open Classes. Finally we investigate performance change w.r.t. the number of open classes. Fig. 11 (c) indicates that our approach demonstrates great robustness to the contamination of open classes.

Further Analysis
In this section, we visualize and interpret memory feature in our framework, as well as present the relation of OLTR++ to fairness analysis and typical failure cases. Infused Memory Feature. In Fig. 13, we inspect the visual concepts that memory feature has infused by visualizing its top activating neurons. Specifically, for each input image, we extracted its top-3 transferred neurons in memory feature. And each neuron is visualized by a collection of highest activated patches [103] over the whole training set. For example, to classify the top left image which belongs to a tail class 'cock', our approach learned to transfer visual concepts that represents "bird head", "round shape" and "dotted texture" respectively. After feature infusion, the dynamic meta-embedding became more informative and discriminative.
We visualize the "memory feature" in Places-LT similarly to ImageNet-LT. Examples of the infused visual concepts from "memory feature" in Places-LT are presented in Fig. 14. We observe that "memory feature" indeed encodes discriminative visual traits for the underlying scene.
Following [27], we visualize the "memory feature" in MS1M-LT by contrasting the least activated average image and the most activated average image of the top firing neuron. From Fig. 15, we observe that "memory feature" in MS1M-LT infused several identity-related attributes (e.g. "high cheekbones", "dark skin color" and "narrow eyes") for precise recognition.
Unsupervised Attribute Discovery. We also quantitatively evaluate the "memory feature" in MS1M-LT by performing unsupervised attribute discovery experiment. The performance is reported on the CelebA dataset [27] via linear probing, as listed in Table 10. Our approach outperforms randomly initialized and ImageNet pretrained features on both top-1 and top-5 average accuracy.
Failure Cases. Since our approach encourages the feature infusion among classes, it slightly sacrifices the fine-grained discrimination for the promotion of under-representative classes. One typical failure case of our approach is the confusion between manyshot and medium-shot classes. For example, the bottom right image in Fig. 13 was misclassified into 'airplane' because some cross-category traits like "nose shape" and "eye shape" were infused. Feature disentanglement [104] and strong contrastive learning [105] are potential alleviations to this trade-off issue.

Input Image
Input Image Input Image Top-1 Transferred Neuron from Memory Feature  Fig. 15: Examples of the infused visual concepts from "memory feature" in MS1M-LT.

CONCLUSIONS
We introduce the OLTR++ task that learns from natural long-tail open-end distributed data and optimizes the overall accuracy over a balanced test set. We propose an integrated OLTR++ algorithm, dynamic meta-embedding, in order to share visual knowledge between head and tail classes and to reduce confusion between tail and open classes. We validated our method on three curated largescale OLTR++ benchmarks (ImageNet-LT, Places-LT and MS1M-LT) as well as active exploration. Our work can enable future researches that are directly transferable to real-world applications.