We investigate a variant of variational autoencoders where there is a superstructure of discrete latent variables on top of the latent features. In general, our superstructure is a tree structure of multiple super latent variables and it is automatically learned from data. When there is only one latent variable in the superstructure, our model reduces to one that assumes the latent features to be generated from a Gaussian mixture model. We call our model the latent tree variational autoencoder (LTVAE). Whereas previous deep learning methods for clustering produce only one partition of data, LTVAE produces multiple partitions of data, each being given by one super latent variable. This is desirable because high dimensional data usually have many different natural facets and can be meaningfully partitioned in multiple ways. Clustering is a fundamental task in unsupervised machine learning, and it is central to many datadriven application domains. Cluster analysis partitions all the data into disjoint groups, and one can understand the structure of the data by examining examples in each group. Many clustering methods have been proposed in the literature BID0 , such as k-means BID18 , Gaussian mixture models BID5 and spectral clustering BID30 . Conventional clustering methods are generally applied directly on the original data space. However, it is challenging to perform cluster analysis on high dimensional and unstructured data BID26 , such as images. It is not only because the dimensionality is high, but also because the original data space is too complex to interpret, e.g. there are semantic gaps between pixel values and objects in images.Recently, deep learning based clustering methods have been proposed that simultanously learn nonlinear embeddings through deep neural networks and perform cluster analysis on the embedding space. The representation learning process learns effective high-level representations from high dimensional data and helps the cluster analysis. This is typically achieved by unsupervised deep learning methods, such as restricted Boltzmann machine (RBM) BID11 , autoencoders (AE) BID28 , variational autoencoders (VAE) BID16 , etc. Previous deep learning based clustering methods BID33 BID10 BID14 BID34 ) assume one single partition over the data and that all attributes define that partition. In real-world applications, however, the assumptions are usually not true. High-dimensional data are often multifaceted and can be meaningfully partitioned in multiple ways based on subsets of attributes BID4 . For example, a student population can be clustered in one way based on course grades and in another way based on extracurricular activities. Movie reviews can be clustered based on both sentiment (positive or negative) and genre (comedy, action, war, etc.) . It is challenging to discover the multi-facet structures of data, especially for high-dimensional data.To resolve the above issues, we propose an unsupervised learning method, latent tree variational autoencoder (LTVAE) to learn latent superstructures in variational autoencoders, and simultaneously perform representation learning and structure learning. LTVAE is a generative model, where the data is assumed to be generated from latent features through neural networks, while the latent features themselves are generated from tree-structured Bayesian networks with another level of latent variables as shown in Fig. 1 . Each of those latent variables defines a facet of clustering. The proposed method automatically selects subsets of latent features for each facet, and learns the dependency structure among different facets. This is achieved through systematic structure learning. Consequently, LTVAE is able to discover complex structures of data rather than one partition. We also propose efficient learning algorithms for LTVAE with gradient descent and Stepwise EM through message passing.The rest of the paper is organized as follows. The related works are reviewed in Section 2. We introduce the proposed method and learning algorithms in Section 3. In Section 4, we present the empirical results. The conclusion is given in Section 5. LTVAE learns the dependencies among latent variables Y. In general, latent variables are often correlated. For example, the social skills and academic skills of a student are generally correlated. Therefore, its better to model this relationship to better fit the data. Experiments show that removing such dependencies in LTVAE models results in inferior data loglikelihood.In this paper, for the inference network, we simply use mean-field inference network with same structure as the generative network BID16 . However, the limited expressiveness of the mean-field inference network could restrict the learning in the generative network and the quality of the learned model BID31 BID6 . Using a faithful inference network structure as in BID31 to incorporate the dependencies among latent variables in the posterior, for example one parameterized with masked autoencoder distribution estimator (MADE) model BID8 , could have a significant improvement in learning. We leave it for future investigation. In this paper, we propose an unsupervised learning method, latent tree variational autoencoder (LT-VAE), which simultaneously performs representation learning and multidimensional clustering. Different from previous deep learning based clustering methods, LTVAE learns latent embeddings from data and discovers multi-facet clustering structure based on subsets of latent features rather than one partition over data. Experiments show that the proposed method achieves state-of-the-art clustering performance and reals reasonable multifacet structures of the data.
We investigate a variant of variational autoencoders where there is a superstructure of discrete latent variables on top of the latent features.
{"src": "scitldr-AIC_train", "paper_id": "SJgNwi09Km"}