Most approaches in generalized zero-shot learning rely on cross-modal mapping between an image feature space and a class embedding space or on generating artificial image features. However, learning a shared cross-modal embedding by aligning the latent spaces of modality-specific autoencoders is shown to be promising in (generalized) zero-shot learning. While following the same direction, we also take artificial feature generation one step further and propose a model where a shared latent space of image features and class embeddings is learned by aligned variational autoencoders, for the purpose of generating latent features to train a softmax classifier. We evaluate our learned latent features on conventional benchmark datasets and establish a new state of the art on generalized zero-shot as well as on few-shot learning. Moreover, our results on ImageNet with various zero-shot splits show that our latent features generalize well in large-scale settings. Generalized zero-shot learning (GZSL) is a classification task where no labeled training examples are available from some of the classes. Many approaches learn a mapping between images and their class embeddings BID11 BID19 BID0 . For instance, ALE maps CNN features of images to a perclass attribute space. An orthogonal approach to GZSL is to augment data by generating artificial image features, such as BID21 who proposed to generate image features via a conditional WGAN. As a third approach, BID16 proposed to learn a latent space embedding by transforming both modalities to the latent spaces of autoencoders and match the corresponding distributions by minimizing the Maximum Mean Discrepancy (MMD). Learning such cross-modal embeddings can be beneficial for potential downstream tasks that require multimodal fusion. In this regard, BID13 recently used a cross-modal autoencoder to extend visual question answering to previously unseen objects.Although recent cross-modal autoencoder architectures represent class prototypes in a latent space BID10 BID16 , better generalization can be achieved if the shared representation space is more amenable to interpolation between different classes. Variational Autoencoders (VAEs) are known for their capability in accurate interpolation between representations in their latent space, i.e. as demonstrated for sentence interpolation BID2 and image interpolation BID6 . Hence, in this work, we train VAEs to encode and decode features from different modalities, and align their latent spaces by matching the parametrized latent distributions and by enforcing a cross-modal reconstruction criterion. Since we learn representations that are oblivious to their origin, a zero-shot visual classifier can be trained using latent space features from semantic data.Our contributions in this work are as follows. FORMULA1 Generalized Zero-shot Learning Let S = {(x, y, c(y))| x ∈ X, y ∈ Y S , c(y) ∈ C} be a set of training examples, consisting of image-features x, e.g. extracted by a CNN, class labels y available during training and class-embeddings c(y). Typical class-embeddings are vectors of continuous attributes or Word2Vec features . In addition, an set U = {(u, c(u))| u ∈ Y u , c(u) ∈ C} is used, where u denote unseen classes from a set Y u , which is disjoint from Y S . Here, C(U ) = {c(u 1 ), ..., c(u L )} is the set of class-embeddings of unseen classes. In the legacy challenge of ZSL, the task is to learn a classifier f ZSL : X → Y U . However, in this work, we focus on the more realistic and challenging setup of generalized zero-shot learning (GZSL) where the aim is to learn a classifier DISPLAYFORM0 The Objective Function CADA-VAE is trained with pairs of image features and attribute vectors of seen classes. The data of each pair has to belong to the same class. In this process, an image feature encoder and an attribute encoder learn to transform the training data into the shared latent space. The encoders belong to two VAEs with a common latent space. Once the VAEs are trained, a softmax classifier is trained on both seen image data and unseen attributes, after they are transformed into the latent representation. As the VAE encoding is non-deterministic , many latent features are sampled for each datapoint. Since we only have one attribute vector per class, we oversample latent-space encoded features of unseen classes. To test the classifier, the visual test data is first transformed into the latent space, using only the predicted means µ of the latent representation. The Objective function for training the VAEs is derived as follows. For every modality i (image features, attributes ), a VAE is trained. The basic VAE loss for a feature x of modality i ∈ 1, 2, ..M is: DISPLAYFORM1 where D KL represents the Kullback-Leibler Divergence, β is a weight, q(z|x (i) ) = N (µ, Σ) is the VAE encoder consisting of a multilayer perceptron, and p(z) is a Gaussian prior. Additionally, each encoded datapoint is decoded into every available modality, e.g. encoded image features are decoded into attributes and vice versa. Consequently, we minimize the L1 cross-reconstruction loss: DISPLAYFORM2 where γ is a weight. The L1 loss empirically proved to provide slighthly better results than L2. Furthermore, the 2-Wasserstein W distance between the multivariate Gaussian latent distribution of image features and attributes is minimized: DISPLAYFORM3 The VAE is trained using the final objective L = L basic +L CA +L DA . We refer to the Cross-Aligned and Distribution-Aligned VAE as CADA-VAE. In addition, we test the variant L = L basic + L CA , termed CA-VAE, and the variant L = L basic + L DA , referred to as DA-VAE. A latent size of 64 is used for all experiments, except 128 for ImageNet. In this work, we propose CADA-VAE, a cross-modal embedding framework for generalized zeroshot learning in which the modality-specific latent distributions are aligned by minimizing their Wasserstein distance and by using cross-reconstruction. This procedure leaves us with encoders that can encode features from different modalities into one cross-modal embedding space, in which a linear softmax classifier can be trained. We present different variants of cross-aligned and distribution aligned VAEs and establish new state-of-the-art results in generalized zero-shot learning for four medium-scale benchmark datasets as well as the large-scale ImageNet. We further show that a cross-modal embedding model for generalized zero-shot learning achieves better performance than data-generating methods, establishing the new state of the art.
We use VAEs to learn a shared latent space embedding between image features and attributes and thereby achieve state-of-the-art results in generalized zero-shot learning.
{"src": "scitldr-AIC_train", "paper_id": "BkghJoRNO4"}