Computer Science
Computer Science-2021.gz:143132
Computer Science
26147e2b3f82417b56f61ee01b02979396b7e1b7
Isometric Propagation Network for Generalized Zero-shot Learning

Zero-shot learning (ZSL) aims to classify images of an unseen class only based on a few attributes describing that class but no access to any training sample. A popular strategy is to learn a mapping between the semantic space of class attributes and the visual space of images based on the seen classes and their data. Thus, an unseen class image can be ideally mapped to its corresponding class attributes. The key challenge is how to align the representations in the two spaces. For most ZSL settings, the attributes for each seen/unseen class are only represented by a vector while the seen-class data provide much more information. Thus, the imbalanced supervision from the semantic and the visual space can make the learned mapping easily overfitting to the seen classes. To resolve this problem, we propose Isometric Propagation Network (IPN), which learns to strengthen the relation between classes within each space and align the class dependency in the two spaces. Specifically, IPN learns to propagate the class representations on an auto-generated graph within each space. In contrast to only aligning the resulted static representation, we regularize the two dynamic propagation procedures to be isometric in terms of the two graphs' edge weights per step by minimizing a consistency loss between them. IPN achieves state-of-the-art performance on three popular ZSL benchmarks. To evaluate the generalization capability of IPN, we further build two larger benchmarks with more diverse unseen classes and demonstrate the advantages of IPN on them.


INTRODUCTION
One primary challenge on the track from artificial intelligence to human-level intelligence is to improve the generalization capability of machine learning models to unseen problems. While most supervised learning methods focus on generalization to unseen data from training task/classes, zeroshot learning (ZSL) (Larochelle et al., 2008;Lampert et al., 2014;Xian et al., 2019a) has a more ambitious goal targeting the generalization to new tasks and unseen classes. In the context of image classification, given images from some training classes, ZSL aims to classify new images from unseen classes with zero training image available. Without training data, it is impossible to directly learn a mapping from the input space to the unseen classes. Hence, recent works introduced learning to map between the semantic space and visual space (Zhu et al., 2019a;Li et al., 2017;Jiang et al., 2018) so that the query image representation and the class representation can be mapped to a shared space for comparison.
However, learning to align the representation usually leads to overfitting on seen classes (Liu et al., 2018). One of the reasons is that in most zero-shot learning settings, the number of images per class is hundreds while the semantic information provided for one class is only limited to one vector. Thus, the mapping function is easily overfitting to the seen classes when trained using the small set of attributes. The learned model typically has imbalanced performance on samples from the seen classes and unseen classes, i.e., strong when predicting samples from the seen classes but struggles in the prediction for samples in the unseen classes.
In this paper, we propose "Isometric Propagation Network (IPN)" which learns to dynamically interact between the visual space and the semantic space. Within each space, IPN uses an attention module to generate a category graph, and then perform multiple steps of propagation on the graph. In every step, the distribution of the attention scores generated in each space are regularized to be isometric so that implicitly aligns the relationships between classes in two spaces. Due to different motifs of the generated graphs, the regularizer for the isometry between the two distributions are provided with sufficient training supervisions and can potentially generalize better on unseen classes.
To get more diverse graph motifs, we also apply episodic training which samples different subset of classes as the nodes for every training episode rather than the whole set of seen classes all the time.
To evaluate IPN, we compared it with several state-of-the-art ZSL methods on three popular ZSL benchmarks, i.e., AWA2 (Xian et al., 2019a), CUB (Welinder et al., 2010) and aPY (Farhadi et al., 2009). To test the generalization ability on more diverse unseen classes rather than similar classes to the seen classes, e.g., all animals for AWA2 and all birds for CUB, we also evaluate IPN on two new large-scale datasets extracted from tieredImageNet (Ren et al., 2018), i.e., tieredImageNet-Mixed and tieredImageNet-Segregated. IPN consistently achieves state-of-the-art performance in terms of the harmonic mean of the accuracy on unseen and seen classes. We show the ablation study of each component in our model and visualize the final learned representation in each space in Figure 1. It shows the class representations in each space learned by IPN are evenly distributed and connected smoothly between the seen and unseen classes. The two spaces can also reflect each other.

RELATED WORKS
Most ZSL methods can be categorized as either discriminative or generative. Discriminative methods generate the classifiers for unseen classes from its attributes by learning a mapping from a input image's visual features to the associated class's attributes on training data of seen classes Akata et al., 2015a;b;Romera-Paredes & Torr, 2015;Kodirov et al., 2017;Socher et al., 2013;Cacheux et al., 2019;Li et al., 2019c), or by relating similar classes according to their shared properties, e.g., semantic features (Norouzi et al., 2013) or phantom classes (Changpinyo et al., 2016), etc. In contrast, generative methods employ the recently popular image generation models (Goodfellow et al., 2014) to synthesize images of unseen classes, which reduces ZSL to supervised learning  but usually increases the training cost and model complexity.
In addition, the study of unsupervised learning (Fan et al., 2021), meta learning (Santoro et al., 2016;a;Ding et al., 2020;2021), or architecture design (Dong et al., 2021;Dong & Yang, 2019) can also be used improve the representation ability of visual or semantic features.
Alignment between two spaces in ZSL ; Socher et al. (2013); Zhang et al. (2017) proposed to transform representations either to the visual space or the semantic space for distance for comparison. Li et al. (2017) introduces how to optimize the semantic space so that the two spaces can be better aligned. Jiang et al. (2018) tries to map the class representations from two spaces to another common space. Comparatively, our IPN does not align two spaces by feature transformation but by regularizing the isometry of the intermediate output from two propagation model.
Graph in ZSL Graph structures have already been explored for their benefits to some discriminative ZSL models, e.g., ; Kampffmeyer et al. (2019) learn to generate the weights of a fully-connected classifier for a given class based on its semantic features computed on a predefined semantic graph using a graph convolutional networks (GCN). They require (1) a pre-trained Figure 2: LEFT: The pipeline of IPN. First, we initialize the visual and semantic prototypes from seen classes's data and class attributes. Then, prototype propagation in each space can be performed over multiple steps. Finally, the concatenated prototypes form the support set of a distance based classifier. RIGHT: Isometric propagation. we propagate prototypes of each class to other connected classes on the graph using attention module. The attention score is regularized to be isometric.
CNN whose last fully-connected layer provides the ground truth of the classifiers ZSL aims to generate; and (2) an existing knowledge graph as the input of GCN. By contrast, our method learns to automatically generate the graph and the class prototypes composing the support set of a distancebased classifier. Hence, our method does not require a pre-trained CNN to provide a ground truth for the classifier. In addition, we train a modified graph attention network (GAN) (rather than a GCN) to generate the prototypes, so it learns both the class adjacency and edges by itself and can provide more accurate modeling of class dependency in scenarios when no knowledge graph is provided. Tong et al. (2019) and Xie et al. (2020) also leverage the idea of graph to transform the feature extracted from the CNN while our graph serves as the paths for propagation. It requires an off-the-shelf graph generated by ConceptNet (Speer et al., 2016), while IPN applies prototype propagation and learns to construct the graph in an end-to-end manner.
Attention in ZSL The attention module has been applied for zero-shot learning for localization. Liu et al. (2019c) introduces localization on the semantic information and Xie et al. (2019) proposes localization on the visual feature. Zhu et al. (2019c) further extends it to multi-localization guided by semantic. Different from them, our attention serves as generating the weights for propagation.
Our episodic training strategy is inspired by the meta-learning method from (Santoro et al., 2016). Similar methods have been broadly studied in meta-learning and few-shot learning approaches (Finn et al., 2017;Snell et al., 2017;Sung et al., 2018) but rarely studied for ZSL in previous works.
In this paper, we show that training ZSL model as a meta-learner can improve the generalization performance and mitigate overfitting.

PROBLEM SETTING
We consider the following setting of ZSL. The training set is composed of data-label pairs (x, y), where x ∈ X is a sample from class y, and y must be one of the seen classes in Y seen . During the test stage, given a test sample x of class y , the ZSL model achieved in training is expected to produce the correct prediction of y. In order to relate the unseen classes in test to the seen classes in training, we are also given the semantic embedding s y of each class y ∈ Y seen ∪ Y unseen . We further assume that seen and unseen classes are disjoint, i.e., Y seen ∩ Y unseen = ∅. The test samples come from both Y seen and Y unseen in generalized zero-shot learning.

ISOMETRIC PROPAGATION NETWORK
In this paper, we bridge the two spaces via class prototypes in both the semantic and visual spaces, i.e., a semantic prototype P s y and a visual prototype P v y for each class y. In order to generate high-quality prototypes, we develop a learnable scheme called "Isometric Propagation Network (IPN)" to iteratively refine the prototypes by propagating them on a graph of classes so each prototype is dynamically determined by the samples and attributes of its related classes.
In this section, we will first introduce the architecture of IPN. An illustration of IPN is given in Figure 2. Then, we will elaborate the episodic training strategy, in which a consistency loss is minimized for isometric alignment between the two spaces.

PROTOTYPE INITIALIZATION
Given a task T defined on a subset of classes Y T with training data D T , IPN first initializes the semantic and visual prototype for each class y ∈ Y T . The semantic prototype of class y is transformed from its attributes s y . Specifically, we have drawn on the expert modules from (Zhang & Shi, 2019) to initialize the semantic prototypes P s y [0]. In previous works (Snell et al., 2017;Santoro et al., 2016), the visual prototype of a class is usually computed by averaging the features of images from the class. In IPN, for each seen class y ∈ Y T ∩ Y seen , we initialize its visual prototype P v where f cnn (·) is a backbone convolutional neural nets (CNN) extracting features from raw images, and W is a linear transformation aiming to change the dimension of the visual prototype to be the same as semantic prototypes. We argue that W is necessary because (1) the visual feature produced by the CNN is usually in high dimension, and reducing its dimension can save significant computations during propagation; (2) keeping the dimensions of the visual and semantic prototypes the same can avoid the dominance of either of them in the model.
During test, we initialize the visual prototype of each unseen class y ∈ Y T ∩Y unseen as the weighted sum of its K-nearest classes from Y seen in the semantic space 1 , i.e., where W is the same transformation from Eq. (1), and N y is the set of the top-K (K is a hyperparameter) seen classes 2 with the largest similarities c s (P s y [0], P s z [0]) to class y in semantic space.

CATEGORY GRAPH GENERATION
Although the propagation needs to be conducted on a category graph, IPN does not rely on any given graph structure as many previous works (Kampffmeyer et al., 2019; so it has less limitations when applied to more general scenarios. In IPN, we inference the category graph of Y T by applying thresholding to the attention scores between the initialized prototypes. In the visual space, the set of edges is defined by where is the threshold. Thereby, the category graph is generated as

ISOMETRIC PROPAGATION
Given the initialized prototypes and generated category graphs in the two spaces, we apply the following prototype propagation in each space for τ steps in order to refine the prototype of each class. In the visual space, each propagation step updates In the semantic space, we apply similar propagation steps over classes on graph G T,s by using the attention module associated with a s (·, ·). Intuitively, the attention module computes the similarity between the two prototypes. In the visual space, given the prototypes P v y and P v z for class y and z, the attention module first computes the following cosine similarity: where h v (·) is a learnable transformation and · 2 represents the 2 norm. For a class y on a category graph, we can use the attention module to aggregate messages from its neighbor classes N y . In this case, we need to further normalize the similarities of y to each z ∈ N y as follows.
where γ is a temperature parameter controlling the smoothness of the normalization. In the semantic space, we have similar definitions of h s (·), c s (·, ·) and a s (·, ·). In the following, we will use P v y [t] and P s y [t] to represent the visual and semantic prototype of class y at the t th step of propagation. We will use | · | to represent the size of a set and ReLU(·) max{·, 0}.
After τ steps of propagation, we concatenate the prototypes achieved in the two spaces for each class to form the final prototypes: The formulation of a s (·, ·) and c s (·, ·) follows the same way as a v (·, ·) and c v (·, ·). To be simple, we don't rewrite the formulations here.

CLASSIFIER GENERATED FROM PROTOTYPES
Given the final prototypes and a query/test image x, we apply a fully-connected layer to predict the similarity between x and each class's prototype. For each class y ∈ Y T , it computes Intuitively, the above classifier can be interpreted as a distance classifier with distance metric f (·, ·). When applied to a query image x, the predicted class is the most probable class, i.e., y = arg max y∈Y T Pr(y|x; P ), where Pr denotes the probability.

TRAINING STRATEGIES
Episodic training on subgraphs. In each episode, we sample a subset of training classes Y T ⊆ Y seen , and then build a training (or support) set D T train and a validation (or query) set D T valid from the training samples of Y T . They are two disjoint sets of samples from the same subset of training classes. We only use D T train to generate the prototypes P by the aforementioned procedures, and then apply backpropagation to minimize the loss of D T valid . This equals to an empirical risk minimization, i.e., This follows the same idea of meta-learning Ravi & Larochelle (2016): it aims to minimize the validation loss instead of the training loss and thus can further improve generalization and restrain overfitting. Moreover, since we train the IPN model on multiple random tasks T and optimize the propagation scheme over different subgraphs (each generated for a random subset of classes), the generalization capability of IPN to unseen classes and new tasks can be substantially enhanced by this meta-training strategy. Moreover, it effectively reduces the propagation costs since computing the attention needs O(|Y T | 2 ) operations. In addition, as shown in the experiments, episodic training is effective in mitigating the problem of data imbalance and overfitting to seen classes, which harasses many ZSL methods.
Consistency loss for alignment of visual and semantic space. In ZSL, the interaction and alignment between different spaces can provide substantial information about class dependency and is critical to the generation of accurate class prototypes. In IPN, we apply prototype propagation in two separate spaces on their own category graphs by using two untied attention modules. In order to keep the propagation in the two spaces consistent, we add an extra consistency loss to the training objective in Eq. (9). It computes the KL-divergence between two distributions defined in the two spaces for each class y ∈ Y T at every propagation step t: where The consistency loss enforces the two attention modules in the two spaces to generate the same attention scores over classes, even when the prototypes and attention module's parameter are different in the two spaces. In this way, they share similar class dependency over the propagation steps. So the generated dual prototypes are well aligned and can provide complementary information when generating the classifier. The final optimization we used to train IPN becomes: where λ is weight for the consistency loss, and P , p v y [t], and p s y [t] are computed only based on D T train , the training (or support) set of task-T for classes in Y T .

EXPERIMENTS
We evaluate IPN and compare it with several state-of-the-art ZSL models in both ZSL setting (where the test set is only composed of unseen classes) and generalized ZSL setting (where the test set is composed of both seen and unseen classes). We report three standard ZSL evaluation metrics, and provide ablation study of several variants of IPN in the generalized ZSL setting. The visualization of the learned class representation is shown in Figure 1.

DATASETS AND EVALUATION CRITERIA
Small and Medium benchmarks. Our evaluation includes experiments on three standard benchmark datasets widely used in previous works, i.e., AWA2 (Xian et al., 2019a), CUB (Welinder et al., 2010) and aPY (Farhadi et al., 2009). The former two are extracted from ImageNet-1K, where AWA2 contains 50 animals classes and CUB contains 200 bird species. aPY includes 32 classes. Detailed statistics of them can be found in Table 1.
Two new large-scale benchmarks. The three benchmark datasets above might be too small for modern DNN models since the training may suffers from large variance and end-to-end training usually fails. Previous ZSL methods therefore fine-tune a pre-trained CNN to achieve satisfactory performance on these datasets, which is costly and less practical. Given these issues, we proposed two new large-scale datasets extracted from tieredImageNet (Ren et al., 2018), i.e., tieredImageNet-Segregated and tieredImageNet-Mixed, whose statistics are given in Table 1. Comparing to the three benchmarks, they are over 10× larger, and training a ResNet101 (He et al., 2016) from scratch on their training sets can achieve a validation accuracy of 65%, indicating that end-to-end training without fine-tuning is possible. These two benchmarks differ in that the "Mixed" version allows a seen class and an unseen class to belong to the same super category, while the "Segregated" version does not. The class attributes are the word embeddings of the class names provided by GloVe (Pennington et al., 2014). We removed the classes that do not have embeddings in GloVe.
Following the most popular setting of ZSL (Xian et al., 2019a), we report the per-class accuracy and harmonic mean of the accuracy on seen and unseen classes.

Dataset #Attributes #Seen #Unseen #Imgs(Tr-S) #Imgs(Te-S) #Imgs(Te-U)
CUB (Welinder et al., 2010)  tieredImageNet-Segregated/Mixed are two new large-scale ones proposed by us with different discrepancy between seen and unseen classes. "Tr-S", "Te-S" and "Te-U" denote seen classes in training, seen classes in test and unseen classes in test.  where "S" denotes the per-class accuracy (%) on seen classes, "U" denotes the per-class accuracy (%) on unseen classes and "H" denotes their harmonic mean.

IMPLEMENTATION AND HYPERPARAMETER DETAILS
For a fair comparison, we follow the setting in (Xian et al., 2019a) and use a pre-trained ResNet-101 (He et al., 2016) to extract 2048-dimensional image features without fine-tuning. For the three standard benchmarks, the model is pre-trained on ImageNet-1K. For the two new datasets, it is pre-trained on the training set used to train ZSL models.
All hyperparameters were chosen on the validation sets provided by Xian et al. (2019a). We use the same hyperparameters tuned on AWA2 for other datasets since they are of similar data type, except for aPY on which we choose learning rate 1.0 × 10 −3 and weight decay 5.0 × 10 −4 . For the rest of the datasets, we train IPN by Adam (Kingma & Ba, 2015) for 360 epochs with a weight decay factor 1.0 × 10 −4 . The learning rate starts from 2.0 × 10 −5 and decays by a multiplicative factor 0.1 every 240 epochs. In each epoch, we update IPN on multiple N -way-K-shot tasks (N -classes with K samples per class), each corresponding to an episode mentioned in Section 4.5. In our experiments, the number of episodes in each epoch is n /NK, where n is the total size of the training set, N = 30 and K = 1 (a small K can mitigate the imbalance between seen and unseen classes); h v and h s are linear transformations; we set temperature γ = 10, threshold = cos 40 o for the cosine similarities between class prototypes, and weight for consistency loss λ = 1. We use propagation steps τ = 2.

MAIN RESULTS
The results are reported in Table 2. Although most previous works (Xian et al., 2019b;a;Sariyildiz & Cinbis, 2019) show that the more complicated generative models usually outperforms the discriminative ones. Our experiments imply that IPN, as a simple discriminative model, can significantly outperform the generative models requiring much more training efforts. Most baselines achieve a high ACC seen but much lower ACC unseen since they suffer from and more sensitive to the imbalance between seen and unseen classes. In contrast, IPN always achieves the highest H-metric among all methods, indicating less overfitting to seen classes. This is due to the more accurate prototypes achieved by iterative propagation in two interactive spaces across seen and unseen classes, and the episodic training strategy, which effectively improves the generalization.
In scaling the experiments to the two large datasets, discriminative models perform better than generative models, which are easily prone to overfitting or divergence. Moreover, during the test stage, generative models require to synthesize data for unseen classes, which may lead to expensive computations when applied to tasks with many classes. In contrast, IPN scales well to these large datasets, outperforming state-of-the-art baselines by a large margin. The performance of most models on tieredImageNet-Segregated is generally worse than tieredImageNet-Mixed because that the unseen classes in the former do not share the same ancestor classes with the seen classes, while in the latter such sharing is allowed.

ABLATION STUDY
Are two prototypes necessary? We compare IPN with its two variants, each using only one out of the two types of prototypes from the two spaces when generating the classifier. The results are shown in Figure 3. It shows that semantic prototype (only) performs much better than visual prototype (only), especially on the unseen classes. This is because that the unseen classes' semantic embedding is given while its visual information is missing (zero images) during the test stage. Although IPN is capable of generating a visual prototype based on other classes' images and class dependency, it is much less accurate than the semantic prototype. However, it also shows that the visual prototypes (even the ones of unseen classes) can provide extra information and further improve the performance of the semantic prototype, e.g., they improve the unseen classes' accuracy from 61.1% to 67.5%. One possible reason is that the visual prototype bridges the query image's visual feature with the semantic features of classes, and thus provides an extra "visual context" for the localization of the correct class. In addition, we observe that IPN with only the visual prototype converges faster than IPN with only a semantic prototype (30 epochs vs. 200 epochs). This might be an artifact caused by the pre-trained CNN, which already produces stable visual prototypes since the early stages. However, it raises a caveat that heavily relying on a pre-trained model might discourage the learning and results in a bad local optimum. How's the benefit of propagation? We then evaluate the effects of the propagation scheme in the two spaces. In particular, we compare IPN with its variants that apply the propagation steps in only one of the two spaces, and the variant that does not apply any propagation after initialization. Their performance is reported in Figure 3. Without any propagation, the classification performance heavily relies on pre-trained CNN. Hence, the accuracy of seen classes can be much higher than that of the unseen classes. Compared to IPN without any propagation, applying propagation in either the visual space or the semantic space can improve the accuracy of unseen classes by 2∼3%, while the accuracy on the seen classes stays almost the same. When equipped with dual propagation, we observe a significant improvement of the accuracy of the unseen classes but a degradation on the seen classes. However, they lead to the highest H-metric, which indicates that the dual propaga-tion achieves a desirable trade-off between overfitting on seen classes and generalization to unseen classes.
Effect of Isometric Alignment In IPN, we minimize a consistency loss during training for alignment of the visual and semantic space. Here, we consider two other options for alignment that exchange the attention modules used in the two spaces, i.e., using the visual attention module in the semantic space, or using the semantic attention module in the visual space. We also include the IPN trained without consistency loss or any alignment as a baseline. Their results are reported in Figure 3. Compared to IPN with/without consistency loss, sharing a single attention module between the two spaces suffers from the overfitting problem on the seen classes. This is because that the visual features and semantic embeddings are two different types of features so they should not share the same similarity metric. In contrast, the consistency loss only enforces the output similarity scores of the two attention modules to be close, while the models/functions computing the similarity can be different. Therefore, to produce the same similarity scores, it does not require each class having the same features in the two spaces. Hence, it leads to a much softer regularization. The proposed IPN has some hyperparameters as mentioned in Section 5.2. To analyze the sensitivity of those hyperparameters, such as threshold, temperature, N and K for N-way-K-shot task, propagation steps, etc. We try different values and show the performance on the AWA2 dataset in Figure 4. The temperature γ is not sensitive for the accuracy. The threshold needs to be bigger than cosine(30 o ) to give IPN some space for the filtering. The propagation steps τ is best at 2. Weight decay is a sensitive hyperparameter and this is also found in previous ZSL literature (Zhang & Shi, 2019;Liu et al., 2019d).

CONCLUSIONS
In this work, we propose an Isometric Propagation Network, which learns to relate classes within each space and across two spaces. IPN achieves state-of-the-art performance and we show the importance of each component and visualize the interpretable relationship of the learned class representations. In future works, we will study other possibilities to initialize the visual prototypes of unseen classes in IPN.

B EXPERIMENTS ON ZERO-SHOT LEARNING SETTING
The comparison between IPN to other baselines on the setting of Zero-shot Learning is shown in Table 3. On CUB, AWA2, aPU, tiered-M, and tiered-S, our IPN outperforms other algorithms by a large margin.

C MORE EXPERIMENTS ON LARGE-SCALE DATASETS
In our work, we follow the train/test split suggested by , who proposed to use the 21K ImageNet dataset for zero-shot evaluation. Here, 2-hops and 3-hops represent that the evaluation is over the classes which are 2/3 hops away from ImageNet-1k. Different from experiments on AWA2, we use N =20 and K=5 for the full ImageNet. Besides, since the edges between each class are provided on the full ImageNet, we directly use them instead of calculating by Eq. (3).