This paper addresses unsupervised few-shot object recognition, where all training images are unlabeled and do not share classes with labeled support images for few-shot recognition in testing. We use a new GAN-like deep architecture aimed at unsupervised learning of an image representation which will encode latent object parts and thus generalize well to unseen classes in our few-shot recognition task. Our unsupervised training integrates adversarial, self-supervision, and deep metric learning. We make two contributions. First, we extend the vanilla GAN with reconstruction loss to enforce the discriminator capture the most relevant characteristics of "fake" images generated from randomly sampled codes. Second, we compile a training set of triplet image examples for estimating the triplet loss in metric learning by using an image masking procedure suitably designed to identify latent object parts. Hence, metric learning ensures that the deep representation of images showing similar object classes which share some parts are closer than the representations of images which do not have common parts. Our results show that we significantly outperform the state of the art, as well as get similar performance to the common episodic training for fully-supervised few-shot learning on the Mini-Imagenet and Tiered-Imagenet datasets. This paper presents a new deep architecture for unsupervised few-shot object recognition. In training, we are given a set of unlabeled images. In testing, we are given a small number K of support images with labels sampled from N object classes that do not appear in the training set (also referred to as unseen classes). Our goal in testing is to predict the label of a query image as one of these N previously unseen classes. A common approach to this N -way K-shot recognition problem is to take the label of the closest support to the query. Thus, our key challenge is to learn a deep image representation on unlabeled data such that it would in testing generalize well to unseen classes, so as to enable accurate distance estimation between the query and support images. Our unsupervised few-shot recognition problem is different from the standard few-shot learning (Snell et al., 2017; Finn et al., 2017) , as the latter requires labeled training images (e.g., for episodic training (Vinyals et al., 2016) ). Also, our problem is different from the standard semi-supervised learning (Chapelle et al., 2009) , where both unlabeled and labeled data are typically allowed to share either all or a subset of classes. When classes of unlabeled and labeled data are different in semi-supervised learning (Chapelle et al., 2009) , the labeled dataset is typically large enough to allow transfer learning of knowledge from unlabeled to labeled data, which is not the case in our few-shot setting. There is scant work on unsupervised few-shot recognition. The state of the art (Hsu et al., 2018) first applies unsupervised clustering (Caron et al., 2018) for learning pseudo labels of unlabeled training images, and then uses the standard few-shot learning on these pseudo labels for episodic traininge.g. , Prototypical Network (Snell et al., 2017) or MAML (Finn et al., 2017) . However, performance of this method is significantly below that of counterpart approaches to supervised few-shot learning. Our approach is aimed at learning an image representation from unlabeled data that captures presence or absence of latent object parts. We expect that such a representation would generalize well We use a GAN-like deep architecture to learn an image encoding z on unlabeled training data that will be suitable for few-shot recognition in testing. Our unsupervised training integrates adversarial, self-supervision, and metric learning. The figure illustrates our first contribution that extends the vanilla GAN (the red dashed line) with regularization so the encodingẑ of a "fake" image is similar to the randomly sampled code z which has been used for generating the "fake" image. The self-supervision task is to predict the rotation angle of rotated real training images. Deep metric learning is illustrated in greater detail in Fig. 3 . to unseen classes in our few-shot recognition task. This is because of the common assumption in computer vision that various distinct object classes share certain parts. Thus, while our labeled and unlabeled images do not show the same object classes, there may be some parts that appear in both training and test image sets. Therefore, an image representation that would capture presence of these common parts in unlabeled images is expected to also be suitable for representing unseen classes, and thus facilitate our N -way K-shot recognition. Toward learning such an image representation, in our unsupervised training, we integrate adversarial, self-supervision, and deep metric learning. As shown in Fig. 1 , we use a GAN-like architecture for training a discriminator network D to encode real images d , which will be later used for few-shot recognition in testing. We also consider a discrete encoding z = D z (x) ∈ {−1, 1} d , and empirically discover that it gives better performance than the continuous counterpart. Hence our interpretation that binary values in the discrete z indicate presence or absence of d latent parts in images. In addition to D z , the discriminator has two other outputs (i.e., heads), D r/f and D rot , for adversarial and self-supervised learning, respectively as illustrated in Fig. 2 . D is adversarially trained to distinguish between real and "fake" images, where the latter x are produced by a generator network G, x = G(z ), from image encodings z which are randomly sampled from the uniform distribution Sampling from the uniform distribution is justified, because latent parts shared among a variety of object classes appearing in the unlabeled training set are likely to be uniformly distributed across the training set. We extend the vanilla GAN with regularization aimed at minimizing a reconstruction loss between the sampled z and the corresponding embeddingẑ = D(G(z )). As our experiments demonstrate, this reconstruction loss plays an important role in training both D and G in combination with the adversarial loss, as both losses enforce G generate as realistic images as possible and D capture the most relevant image characteristics for reconstruction and real/fake recognition. Furthermore, following recent advances in self-supervised learning (Doersch et al., 2015; Zhang et al., 2016; Noroozi & Favaro, 2016; Noroozi et al., 2017; Zhang et al., 2017) , we also augment our training set with rotated versions of the real images around their center, and train D to predict their rotation angles,α = D rot (Rotate(x, α)) ∈ {0, 1, 2, 3} * 90 • . As in other approaches that use self-supervised learning, our results demonstrate that this data augmentation strengthens our unsupervised training and improves few-shot recognition. Finally, we use deep metric learning toward making the image encoding z = D z (x) represent latent parts and in this way better capture similarity of object classes for our few-shot recognition. We expect that various object classes share parts, and that more similar classes have more common parts. Therefore, the encodings of images showing similar (or different) object classes should have a small (or large) distance. To ensure this property, we use metric learning and compile a new training set of triplet images for estimating the standard triple loss, as illustrated in Fig. 3 . Since classes in our training set are not annotated, we form the triplet training examples by using an image masking procedure which is particularly suitable for identifying latent object parts. In the triplet, the anchor is the original (unmasked) image, the positive is an image obtained from the original by masking rectangular patches at the image periphery (e.g., top corner), and the negative is an image obtained from the original by masking centrally located image patches. By design, the negative image masks an important object part, and thus the deep representations of the anchor and the negative should have a large distance. Conversely, masking peripheral corners in the positive image does not cover any important parts of the object, and thus the deep representation of the positive should be very close to that of the anchor. In this way, our metric learning on the triplet training examples ensures that the learned image representation z accounts for similarity of object classes in terms of their shared latent parts. As our results show, this component of our unsupervised training further improves few-shot recognition in testing, to the extent that not only do we significantly outperform the state of the art but also get a performance that is on par with the common episodic training for fullysupervised few-shot learning on the Mini-Imagenet (Vinyals et al., 2016; Ravi & Larochelle, 2016) and Tiered-Imagenet (Ren et al., 2018) datasets. Our contributions are twofold: • Extending the vanilla GAN with a reconstruction loss between uniformly sampled codes, , and embeddings of the corresponding "fake" images,ẑ = D(G(z )). • The masking procedure for compiling triplet image examples and deep metric learning of z so it accounts for image similarity in terms of shared latent parts. The rest of this paper is organized as follows. Sec. 2 reviews previous work, Sec. 3 specifies our proposed approach, Sec. 4 presents our implementation details and our experimental results, and finally, Sec. 5 gives our concluding remarks. We have addressed unsupervised few-shot object recognition, where all training images are unlabeled and do not share classes with test images. A new GAN-like deep architecture has been (Donahue et al., 2016) 25.56 ± 1.08 31.10 ± 0.63 --AAL-ProtoNets (Antoniou & Storkey, 2019) 37.67 ± 0.39 40.29 ± 0.68 --UMTRA + AutoAugment (Khodadadeh et al., 2018) 39.93 50.73 --DeepCluster CACTUs -ProtoNets (Hsu et al., 2018) 39. (Snell et al., 2017) 46.56 ± 0.76 62.29 ± 0.71 46.52 ± 0.72 66.15 ± 0.74 Figure 4: Our image masking with rectangular patches for Mini-Imagenet. In every row, the images are organized from left to right in the descending order by their estimated distance to the original (unmasked) image. proposed for unsupervised learning of an image representation which respects image similarity in terms of shared latent object parts. We have made two contributions by extending the vanilla GAN with reconstruction loss and by integrating deep metric learning with the standard adversarial and self-supervision learning. Our results demonstrate that our approach generalizes will to unseen classes, outperforming the sate of the art by more than 8% in both 1-shot and 5-shot recognition tasks on the benchmark Mini-Imagenet dataset. We have reported the first results of unsupervised few-shot recognition on the Tiered-Imagenet dataset. Our ablations have evaluated that solely our first contribution leads to superior performance relative to that of closely related approaches, and that the addition of the second contribution further improves our 1-shot and 5-shot recognition by 3%. We also outperform a recent fully-supervised approach to few-shot learning that uses the common episodic training on the same datasets.
We address the problem of unsupervised few-shot object recognition, where all training images are unlabeled and do not share classes with test images.
{"src": "scitldr-AIC_train", "paper_id": "SyxhapVYvH"}