A Survey on Context Learning <s> Implicit Analysis + Neural Network <s> In this paper,we introduce a new topic model named Gaussian-LDA, which is more suitable to model continuous data. Topic Model based on latent Dirichlet allocation (LDA) is widely used for the statistical analysis of document collections and other discrete data. The LDA model assumes that the words of each document arise from a mixture of topics, each of which is a multinomial distribution over the vocabulary. To apply the original LDA to process continuous data, discretization based vector quantization must be done beforehand, which usually results in information loss. In the proposed model, we consider continuous emission probability, Gaussian instead of multinomial distribution. This new topic model demonstrates higher performance than standard LDA in the experiments of audio retrieval. <s> BIB001 </s> A Survey on Context Learning <s> Implicit Analysis + Neural Network <s> Continuous space word embeddings learned from large, unstructured corpora have been shown to be effective at capturing semantic regularities in language. In this paper we replace LDA’s parameterization of “topics” as categorical distributions over opaque word types with multivariate Gaussian distributions on the embedding space. This encourages the model to group words that are a priori known to be semantically related into topics. To perform inference, we introduce a fast collapsed Gibbs sampling algorithm based on Cholesky decompositions of covariance matrices of the posterior predictive distributions. We further derive a scalable algorithm that draws samples from stale posterior predictive distributions and corrects them with a Metropolis–Hastings step. Using vectors learned from a domain-general corpus (English Wikipedia), we report results on two document collections (20-newsgroups and NIPS). Qualitatively, Gaussian LDA infers different (but still very sensible) topics relative to standard LDA. Quantitatively, our technique outperforms existing models at dealing with OOV words in held-out documents. <s> BIB002 </s> A Survey on Context Learning <s> Implicit Analysis + Neural Network <s> Latent Dirichlet Allocation (LDA) mining thematic structure of documents plays an important role in nature language processing and machine learning areas. However, the probability distribution from LDA only describes the statistical relationship of occurrences in the corpus and usually in practice, probability is not the best choice for feature representations. Recently, embedding methods have been proposed to represent words and documents by learning essential concepts and representations, such as Word2Vec and Doc2Vec. The embedded representations have shown more effectiveness than LDA-style representations in many tasks. In this paper, we propose the Topic2Vec approach which can learn topic representations in the same semantic vector space with words, as an alternative to probability. The experimental results show that Topic2Vec achieves interesting and meaningful results. <s> BIB003
Many implicit analysis methods and neural network methods are complementary to each other in the sense of context scope. Generally speaking, most implicit analysis methods are based on global statistics and the bag-of-words assumption-the word order in a document is neglected, whereas, most neural network methods are based on local context and takes into consideration the word order, at least in the small context window. This complementary relationship makes them a good combination to incorporate prior knowledge and enhance their performance. Ref. BIB002 proposes a Gaussian LDA model based on word embeddings. Gaussian LDA was first introduced by BIB001 to model continuous data, such as audio signal. In the conventional LDA model, topics are categorical multinomial distributions over discrete word types, whereas in the proposed Gaussian LDA model, topics are replaced with multivariate Gaussian distributions in the vector space of pre-trained word embeddings. Continuous word embeddings learnt from large corpora such as Wikipedia are proven to be effective at capturing semantic regularities in language. This encourages us to use word embeddings as prior knowledge and to group words known to be semantically related into same topics. The first step is to learn word embeddings from large corpora. Documents now are sequences of word embeddings rather than word types. Then the generative process can be described as: For each topic k ¼ 1; 2; . . . ; K: 1) Draw topic covariance S S k $ W À1 ðC C; nÞ. Gaussian LDA uses pre-trained word vectors as prior knowledge and model topics in this continous vector space, while BIB003 provides an opposite idea to combine topic model and neural network model. This paper proposes the Topic2-Vec model which uses pre-trained topic label of each word as the prior knowledge. The first step of Topic2Vec is to learn topic labels for words with LDA, and thus global statistics is contained in this topic information. When training the neural network language model, a set of neurons representing topics are added in the output layer, i.e., in each training step, the prediction task is to infer both the current word and the topic label for the current word based on the context words. As a result, the proposed model can learn distributed representations for topics in the same semantic space with words and produces a good grouping and separation of word vectors in different topics.