Inference Methods for Latent Dirichlet Allocation
Latent Dirichlet Allocation (LDA) has seen a huge number of works surrounding it in recent years in the machine learning and text mining communities. Numerous inference algorithms for the model have been introduced, eachwith its trade o s. In this survey, we investigate some of themain strategies that have been applied to inference in this model and summarize the current state of the art in LDA inference methods. 1 The Dirichlet and its Relation to the Multinomial Before exploring Latent Dirichlet Allocation in depth, it is important to understand some properties of the Dirichlet distribution it uses as a component. The Dirichlet distribution with parameter vector a of length K is de ned as Dirichlet(th ;a) 1 B(a) K i=1 thai 1 i (1) where B(a) is the multivariate Beta function, which can be expressed using the gamma function as B(a) K i=1 G(ai G (K i=1 ai (2) The Dirichlet provides a distribution over vectors x that lie on the (k 1) simplex. This is a complicated way of saying that the Dirichlet is a distribution over vectors th Rk such that the values thi [0, 1] and |th |1 1 (the values in th sum to 1) In other words, the Dirichlet is a distribution over the possible parameter vectors for a Multinomial distribution. This fact is used in Latent Dirichlet allocation (hence its name) to provide a principled way of generating the multinomial distributions that comprise the word distributions for the topics as well as the topic proportions within each document. The Dirichlet distribution, in addition to generating proper parameter vectors for a Multinomial, can be shown to be what is called a conjugate prior to the Multinomial. This means that if one were to use a Dirichlet distribution as a prior over the parameters of a Multinomial distribution, the resulting