A Survey on Context Learning <s> Latent Dirichlet Allocation Based Analysis <s> We describe latent Dirichlet allocation (LDA), a generative probabilistic model for collections of discrete data such as text corpora. LDA is a three-level hierarchical Bayesian model, in which each item of a collection is modeled as a finite mixture over an underlying set of topics. Each topic is, in turn, modeled as an infinite mixture over an underlying set of topic probabilities. In the context of text modeling, the topic probabilities provide an explicit representation of a document. We present efficient approximate inference techniques based on variational methods and an EM algorithm for empirical Bayes parameter estimation. We report results in document modeling, text classification, and collaborative filtering, comparing to a mixture of unigrams model and the probabilistic LSI model. <s> BIB001 </s> A Survey on Context Learning <s> Latent Dirichlet Allocation Based Analysis <s> Statistical approaches to language learning typically focus on either short-range syntactic dependencies or long-range semantic dependencies between words. We present a generative model that uses both kinds of dependencies, and can be used to simultaneously find syntactic classes and semantic topics despite having no representation of syntax or semantics beyond statistical dependency. This model is competitive on tasks like part-of-speech tagging and document classification with models that exclusively use short- and long-range dependencies respectively. <s> BIB002 </s> A Survey on Context Learning <s> Latent Dirichlet Allocation Based Analysis <s> A family of probabilistic time series models is developed to analyze the time evolution of topics in large document collections. The approach is to use state space models on the natural parameters of the multinomial distributions that represent the topics. Variational approximations based on Kalman filters and nonparametric wavelet regression are developed to carry out approximate posterior inference over the latent topics. In addition to giving quantitative, predictive models of a sequential corpus, dynamic topic models provide a qualitative window into the contents of a large document collection. The models are demonstrated by analyzing the OCR'ed archives of the journal Science from 1880 through 2000. <s> BIB003 </s> A Survey on Context Learning <s> Latent Dirichlet Allocation Based Analysis <s> This paper addresses several key issues in the ArnetMiner system, which aims at extracting and mining academic social networks. Specifically, the system focuses on: 1) Extracting researcher profiles automatically from the Web; 2) Integrating the publication data into the network from existing digital libraries; 3) Modeling the entire academic network; and 4) Providing search services for the academic network. So far, 448,470 researcher profiles have been extracted using a unified tagging approach. We integrate publications from online Web databases and propose a probabilistic framework to deal with the name ambiguity problem. Furthermore, we propose a unified modeling approach to simultaneously model topical aspects of papers, authors, and publication venues. Search services such as expertise search and people association search have been provided based on the modeling results. In this paper, we describe the architecture and main features of the system. We also present the empirical evaluation of the proposed methods. <s> BIB004 </s> A Survey on Context Learning <s> Latent Dirichlet Allocation Based Analysis <s> We develop the relational topic model (RTM), a model of documents and the links between them. For each pair of documents, the RTM models their link as a binary random variable that is conditioned on their contents. The model can be used to summarize a network of documents, predict links between them, and predict words within them. We derive efficient inference and learning algorithms based on variational methods and evaluate the predictive performance of the RTM for large networks of scientific abstracts and web documents. <s> BIB005 </s> A Survey on Context Learning <s> Latent Dirichlet Allocation Based Analysis <s> We develop the relational topic model (RTM), a hierarchical model of both network structure and node attributes. We focus on document networks, where the attributes of each document are its words, that is, discrete observations taken from a fixed vocabulary. For each pair of documents, the RTM models their link as a binary random variable that is conditioned on their contents. The model can be used to summarize a network of documents, predict links between them, and predict words within them. We derive efficient inference and estimation algorithms based on variational methods that take advantage of sparsity and scale with the number of links. We evaluate the predictive performance of the RTM for large networks of scientific abstracts, web documents, and geographically tagged news. <s> BIB006 </s> A Survey on Context Learning <s> Latent Dirichlet Allocation Based Analysis <s> We introduce the author-topic model, a generative model for documents that extends Latent Dirichlet Allocation (LDA; Blei, Ng, & Jordan, 2003) to include authorship information. Each author is associated with a multinomial distribution over topics and each topic is associated with a multinomial distribution over words. A document with multiple authors is modeled as a distribution over topics that is a mixture of the distributions associated with the authors. We apply the model to a collection of 1,700 NIPS conference papers and 160,000 CiteSeer abstracts. Exact inference is intractable for these datasets and we use Gibbs sampling to estimate the topic and author distributions. We compare the performance with two other generative models for documents, which are special cases of the author-topic model: LDA (a topic model) and a simple author model in which each author is associated with a distribution over words rather than a distribution over topics. We show topics recovered by the author-topic model, and demonstrate applications to computing similarity between authors and entropy of author output. <s> BIB007
Though PLSI has shown its effectiveness, it has the issue of over-fitting due to the linear growth in the number of parameters with the number of training documents. To address this issue, Latent Dirichlet Allocation BIB001 is proposed. LDA has been ubiquitously implemented as a generative probabilistic model to analyze text corpus. The LDA model also follows the bag-of-word assumption where only the number of appearance for each word in a document is considered. As with PLSI, the model assumes that each topic holds a specific vocabulary distribution, e.g., "gene" and "cell" are highly likely to be produced from the "biology" topic. The generative process of LDA can be formally described as: sample a topic from the topic distribution, z $ Multinomialðu d Þ. sample a word w from a multinomial distribution conditioned on the selected topic, w $ Multinomialðwjz; bÞ. Hence, the LDA model can be represented as Fig. 3 . Given the parameters a and b, the conditional distribution of a document is obtained by integrating the joint distribution over u and z pðwja; bÞ ¼ Z pðujaÞ pðz n juÞpðw n jz n ; bÞdu; where pðujaÞ is the Dirichlet distribution and pðz n juÞ is u i for the unique i such that z i n ¼ 1. This optimization problem can be solved by the variational EM algorithm and Gibbs sampling. In terms of perplexity, LDA achieves an improvement of 30 percent over PLSI and 48 percent over Unigram model BIB001 . Although the conventional LDA can effectively extract topics based on the frequency of words, it ignores the order among the words which matters in some cases. For instance, for protein sequences, the different alignment of amino-acids have different biological meaning. In BIB002 , the authors propose a composite generative model that combines the semantic and syntactic components. While the generative process for the semantic component follows similarly with that of LDA, the syntactic component is mainly responsible for the short-range dependencies and is modeled by an Hidden Markov Model (HMM). Hence the semantic component is based on global context whereas the syntactic component is based local context. The integration of both the semantic and syntactic components enables the factorization of a sentence into function words and content words. More specifically, n syntactic word classes fc 1 ; c 2 ; . . . ; c n g are introduced. Also we utilize p c iÀ1 to denote the transition probability from class c iÀ1 in HMM. In this way the generative process of the composite model can be described as follows: draw a word from semantic component, otherwise draw a word from syntactic component. Ref. BIB007 introduces the author-topic model and discusses how to utilize the author context information when analyzing document collections. In this model, each author is associated with a unique multinomial distribution over words, corresponding to the fact that each author has his or her own interest and domain. Hence, words are drawn according to the author topics. A document with multiple authors is modeled as a mixture distribution of the distributions associated with the authors. When a group of authors a d have written a document d, the process to generate a word is: first, choose the author x responsible for a given word. Author x is associated with a distribution over topics, and accordingly parameter u is chosen from a symmetric Dirichlet distribution. And a topic z is selected based on the author, then a word is generated according to the distribution corresponding to the selected topic. They conduct experiments on the NIPS dataset where papers are extracted from the NIPS conferences between 1987 and 1999. The author-topic model can gain a lower perplexity (approximately 7 percent) than LDA in the early training step, as the contextual author information provides a better prior knowledge for the documents. However as the training goes on, LDA becomes better than the author-topic model because in practice no two research papers are exactly the same. In BIB004 , they furthermore incorporate the conference context information into the model for academic document collections, considering the fact that each research conference has its own topics. Later on, much more LDA variants with local context information have been proposed. Ref. BIB005 proposes the relational topic model for documents with links between them, Ref. BIB006 then introduces a hierarchical architecture upon it. BIB003 utilizes the temporal context information and develops a family of probabilistic time series models named the dynamic topic model to analyze the time evolution of topics in large document collections.