Effectively inferring discriminative and coherent latent topics of short texts is a critical task for many real world applications. Nevertheless, the task has been proven to be a great challenge for traditional topic models due to the data sparsity problem induced by the characteristics of short texts. Moreover, the complex inference algorithm also become a bottleneck for these traditional models to rapidly explore variations. In this paper, we propose a novel model called Neural Variational Sparse Topic Model (NVSTM) based on a sparsity-enhanced topic model named Sparse Topical Coding (STC). In the model, the auxiliary word embeddings are utilized to improve the generation of representations. The Variational Autoencoder (VAE) approach is applied to inference the model efficiently, which makes the model easy to explore extensions for its black-box inference process. Experimental results onWeb Snippets, 20Newsgroups, BBC and Biomedical datasets show the effectiveness and efficiency of the model. With the great popularity of social networks and Q&A networks, short texts have been the prevalent information format on the Internet. Uncovering latent topics from huge volume of short texts is fundamental to many real world applications such as emergencies detection BID18 , user interest modeling BID19 , and automatic query-reply BID16 . However, short texts are characteristic of short document length, a very large vocabulary, a broad range of topics, and snarled noise, leading to much sparse word co-occurrence information. Thus, the task has been proven to be a great challenge to traditional topic models. Moreover, the complex inference algorithm also become a bottleneck for these traditional models to rapidly explore variations.To address the aforementioned issue, there are many previous works introducing new techniques such as word embeddings and neural variational inference to topic models. Word embeddings are the low-dimensional real-valued vectors for words. It have proven to be effective at capturing syntactic and semantic information of words. Recently, many works have tried to incorporate word embeddings into topic models to enrich topic modeling BID5 BID7 BID22 . Yet these models general rely on computationally expensive inference procedures like Markov Chain Monte Carlo, which makes them hard to rapidly explore extensions. Even minor changes to model assumptions requires a re-deduction of the inference algorithms, which is mathematic challenging and time consuming. With the advent of deep neural networks, the neural variational inference has emerged as a powerful approach to unsupervised learning of complicated distributions BID8 BID17 BID14 . It approximates the posterior of a generative model with a variational distribution parameterized by a neural network, which allows back-propagation based function approximations in generative models. The variational autoencoder (VAE) BID8 , one of the most popular deep generative models, has shown great promise in modeling complicated data. Motivated by the promising potential of VAE in building generative models with black-box inference process, there are many works devoting to inference topic models with VAE BID20 BID13 BID4 . However, these methods yield the same poor performance in short texts as LDA.Based on the analysis above, we propose a Neural Variational Sparse Topic Model (NVSTM) based on a sparsity-enhanced topic model STC for short texts. The model is parameterized with neural networks and trained with VAE. It still follows the probabilistic characteristics of STC. Thus, the model inherit s the advantages of both sparse topic models and deep neural networks. Additionally, we exploit the auxiliary word embeddings to improve the generation of short text representations.1. We propose a novel Neural Variational Sparse Topic Model (NVSTM) to learn sparse representations of short texts. The VAE is utilized to inference the model effectively. 2. The general word semantic information is introduced to improve the sparse representations of short texts via word embeddings. 3. We conduct experiments on four datasets. Experimental results demonstrate our model's superiority in topic coherence and text classification accuracy.The rest of this paper is organized as follows. First, we reviews related work. Then, we present the details of the proposed NVSTM, followed by the experimental results. Finally, we draw our conclusions. We propose a neural sparsity-enhanced topic model NVSTM, which is the first effort in introducing effective VAE inference algorithm to STC as far as we know. We take advantage of VAE to simplify the inference process, which require no model-specific algorithm derivations. With the employing of word embeddings and neural network framework, NVSTM is able to generate clearer and semanticenriched representations for short texts. The evaluation results demonstrate the effectiveness and efficiency of our model. Future work can include extending our model with other deep generative models, such as generative adversarial network (GAN).
a neural sparsity-enhanced topic model based on VAE
{"src": "scitldr-AIC_train", "paper_id": "ryQz_ZfHz"}