Learning rich representation from data is an important task for deep generative models such as variational auto-encoder (VAE). However, by extracting high-level abstractions in the bottom-up inference process, the goal of preserving all factors of variations for top-down generation is compromised. Motivated by the concept of “starting small”, we present a strategy to progressively learn independent hierarchical representations from high- to low-levels of abstractions. The model starts with learning the most abstract representation, and then progressively grow the network architecture to introduce new  representations at different levels of abstraction. We quantitatively demonstrate the ability of the presented model to improve disentanglement in comparison to existing works on two benchmark datasets using three disentanglement metrics, including a new metric we proposed to complement the previously-presented metric of mutual information gap. We further present both qualitative and quantitative evidence on how the progression of learning improves disentangling of hierarchical representations. By drawing on the respective advantage of hierarchical representation learning and progressive learning, this is to our knowledge the first attempt to improve disentanglement by progressively growing the capacity of VAE to learn hierarchical representations. Variational auto-encoder (VAE), a popular deep generative model (DGM), has shown great promise in learning interpretable and semantically meaningful representations of data ; Chen et al. (2018) ; Kim & Mnih (2018) ). However, VAE has not been able to fully utilize the depth of neural networks like its supervised counterparts, for which a fundamental cause lies in the inherent conflict between the bottom-up inference and top-down generation process (Zhao et al. (2017) ; Li et al. (2016) ): while the bottom-up abstraction is able to extract high-level representations helpful for discriminative tasks, the goal of generation requires the preservation of all generative factors that are likely at different abstraction levels. This issue was addressed in recent works by allowing VAEs to generate from details added at different depths of the network, using either memory modules between top-down generation layers (Li et al. (2016) ), or hierarchical latent representations extracted at different depths via a variational ladder autoencoder (VLAE, Zhao et al. (2017) ). However, it is difficult to learn to extract and disentangle all generative factors at once, especially at different abstraction levels. Inspired by human cognition system, Elman (1993) suggested the importance of "starting small" in two aspects of the learning process of neural networks: incremental input in which a network is trained with data and tasks of increasing complexity, and incremental memory in which the network capacity undergoes developmental changes given fixed external data and tasks -both pointing to an incremental learning strategy for simplifying a complex final task. Indeed, the former concept of incremental input has underpinned the success of curriculum learning (Bengio et al. (2015) ). In the context of DGMs, various stacked versions of generative adversarial networks (GANs) have been proposed to decompose the final task of high-resolution image generation into progressive sub-tasks of generating small to large images (Denton et al. (2015) ; Zhang et al. (2018) ). The latter aspect of "starting small" with incremental growth of network capacity is less explored, although recent works have demonstrated the advantage of progressively growing the depth of GANs for generating high-resolution images (Karras et al. (2018) ; ). These works, so far, have focused on progressive learning as a strategy to improve image generation. We are motivated to investigate the possibility to use progressive learning strategies to improve learning and disentangling of hierarchical representations. At a high level, the idea of progressively or sequentially learning latent representations has been previously considered in VAE. In Gregor et al. (2015) , the network learned to sequentially refine generated images through recurrent networks. In Lezama (2019) , a teacher-student training strategy was used to progressively increase the number of latent dimensions in VAE to improve the generation of images while preserving the disentangling ability of the teacher model. However, these works primarily focus on progressively growing the capacity of VAE to generate, rather than to extract and disentangle hierarchical representations. In comparison, in this work, we focus on 1) progressively growing the capacity of the network to extract hierarchical representations, and 2) these hierarchical representations are extracted and used in generation from different abstraction levels. We present a simple progressive training strategy that grows the hierarchical latent representations from different depths of the inference and generation model, learning from high-to low-levels of abstractions as the capacity of the model architecture grows. Because it can be viewed as a progressive strategy to train the VLAE presented in Zhao et al. (2017) , we term the presented model pro-VLAE. We quantitatively demonstrate the ability of pro-VLAE to improve disentanglement on two benchmark data sets using three disentanglement metrics, including a new metric we proposed to complement the metric of mutual information gap (MIG) previously presented in Chen et al. (2018) . These quantitative studies include comprehensive comparisons to β-VAE ), VLAE (Zhao et al. (2017) ), and the teacher-student strategy as presented in (Lezama (2019) ) at different values of the hyperparameter β. We further present both qualitative and quantitative evidence that pro-VLAE is able to first learn the most abstract representations and then progressively disentangle existing factors or learn new factors at lower levels of abstraction, improving disentangling of hierarhical representations in the process. In this work, we present a progressive strategy for learning and disentangling hierarchical representations. Starting from a simple VAE, the model first learn the most abstract representation. Next, the model learn independent representations from high-to low-levels of abstraction by progressively growing the capacity of the VAE deep to shallow. Experiments on several benchmark data sets demonstrated the advantages of the presented method. An immediate future work is to include stronger guidance for allocating information across the hierarchy of abstraction levels, either through external multi-scale image supervision or internal information-theoretic regularization strategies.
We proposed a progressive learning method to improve learning and disentangling latent representations at different levels of abstraction.
{"src": "scitldr-AIC_train", "paper_id": "SJxpsxrYPS"}