Unsupervised representation learning holds the promise of exploiting large amount of available unlabeled data to learn general representations. A promising technique for unsupervised learning is the framework of Variational Auto-encoders (VAEs). However, unsupervised representations learned by VAEs are significantly outperformed by those learned by supervising for recognition. Our hypothesis is that to learn useful representations for recognition the model needs to be encouraged to learn about repeating and consistent patterns in data. Drawing inspiration from the mid-level representation discovery work, we propose PatchVAE, that reasons about images at patch level. Our key contribution is a bottleneck formulation in a VAE framework that encourages mid-level style representations. Our experiments demonstrate that representations learned by our method perform much better on the recognition tasks compared to those learned by vanilla VAEs. Due to the availability of large labeled visual datasets, supervised learning has become the dominant paradigm for visual recognition. That is, to learn about any new concept, the modus operandi is to collect thousands of labeled examples for that concept and train a powerful classifier, such as a deep neural network. This is necessary because the current generation of models based on deep neural networks require large amounts of labeled data (Sun et al., 2017) . This is in stark contrast to the insights that we have from developmental psychology on how infants develop perception and cognition without any explicit supervision (Smith & Gasser, 2005) . Moreover, the supervised learning paradigm is ill-suited for applications, such as health care and robotics, where annotated data is hard to obtain either due to privacy concerns or high cost of expert human annotators. In such cases, learning from very few labeled images or discovering underlying natural patterns in large amounts of unlabeled data can have a large number of potential applications. Discovering such patterns from unlabeled data is the standard setup of unsupervised learning. Over the past few years, the field of unsupervised learning in computer vision has followed two seemingly different tracks with different goals: generative modeling and self-supervised learning. The goal of generative modeling is to learn the probability distribution from which data was generated, given some training data. A learned model can draw samples from the same distribution or evaluate the likelihoods of new data. Generative models are also useful for learning compact representation of images. However, we argue that these representations are not as useful for visual recognition. This is not surprising since the task of reconstructing images does not require the bottleneck representation to sort out meaningful data useful for recognition and discard the rest; on the contrary, it encourages preserving as much information as possible for reconstruction. In comparison, the goal in selfsupervised learning is to learn representations that are useful for recognition. The standard paradigm is to establish proxy tasks that don't require human-supervision but can provide signals useful for recognition. Due to the mismatch in goals of unsupervised learning for visual recognition and the representations learned from generative modeling, self-supervised learning is a more popular way of learning representations from unlabeled data. However, fundamental limitation of this self-supervised paradigm is that we need to define a proxy-task that can mimic the desired recognition. It is not always possible to establish such a task, nor are these tasks generalizable across recognition tasks. In this paper, we take the first steps towards enabling the unsupervised generative modeling approach of VAEs to learn representations useful for recognition. Our key hypothesis is that for a representation to be useful, it should capture just the interesting parts of the images, as opposed to everything in the images. What constitutes an interesting image part has been defined and studied in earlier works that pre-date the end-to-end trained deep network methods Doersch et al., 2012; Juneja et al., 2013) . Taking inspiration from these works, we propose a novel representation that only encodes such few parts of an image that are repetitive across the dataset, i.e., the patches that occur often in images. By avoiding reconstruction of the entire image our method can focus on regions that are repeating and consistent across many images. In an encoder-decoder based generative model, we constrain the encoder architecture to learn such repetitive parts -both in terms of representations for appearance of these parts (or patches in an image) and where these parts occur. We formulate this using variational auto-encoder (β-VAEs) (Kingma & Welling, 2013; Matthey et al., 2017) , where we impose novel structure on the latent representations. We use discrete latents to model part presence or absence and continuous latents to model their appearance. We present this approach, PatchVAE, in Section 3 and demonstrate that it learns representations that are much better for recognition as compared to those learned by the standard β-VAEs (Kingma & Welling, 2013; Matthey et al., 2017) . In addition, we propose in Section 3.4 that losses that favor foreground, which is more likely to contain repetitive patterns, result in representations that are much better at recognition. In Section 4, we present results on CIFAR100 (Krizhevsky et al., 2009) , MIT Indoor Scene Recognition (Quattoni & Torralba, 2009) , Places (Zhou et al., 2017) , and ImageNet (Deng et al., 2009 ) datasets. Our contributions are as follows: • We propose a novel patch-based bottleneck in the VAE framework that learns representations that can encode repetitive parts across images. • We demonstrate that our method, PatchVAE, learns unsupervised representations that are better suited for recognition in comparison to traditional VAEs. • We show that losses that favor foreground are better for unsupervised learning of representations for recognition. • We perform extensive ablation analysis to understand the importance of different aspects of the proposed PatchVAE architecture. We presented a patch-based bottleneck in a VAE framework that encourages learning useful representations for recognition. Our method, PatchVAE, constrains the encoder architecture to only learn patches that are repetitive and consistent in images as opposed to learning everything, and therefore results in representations that perform much better for recognition tasks compared to vanilla VAEs. We also demonstrate that losses that favor high-energy foreground regions of an image are better for unsupervised learning of representations for recognition. 6 APPENDIX 6.1 VISUALIZATION OF WEIGHTED LOSS Figure 3 shows an illustration of the reconstruction loss L w proposed in Section 3.4. Notice that in first column, guitar has more weight that rest of the image. Similarly in second, fourth and sixth columns that train, painting, and people are respectively weighed more heavily by L w than rest of the image; thus favoring capturing the foreground regions.
A patch-based bottleneck formulation in a VAE framework that learns unsupervised representations better suited for visual recognition.
{"src": "scitldr-AIC_train", "paper_id": "r1x1kJHKDH"}