Probabilistic Latent Semantic Analysis ( PLSA ) models have been shown to provide a better model for capturing polysemy and synonymy than Latent Semantic Analysis ( LSA ) . However , the parameters of a PLSA model are trained using the Expectation Maximization ( EM ) algorithm , and as a result , the trained model is dependent on the initialization values so that performance can be highly variable . In this paper we present a method for using LSA analysis to initialize a PLSA model . We also investigated the performance of our method for the tasks of text segmentation and retrieval on personal-size corpora , and present results demonstrating the efficacy of our proposed approach . In modeling a collection of documents for information access applications , the documents are often represented as a " bag of words " , i.e. , as term vectors composed of the terms and corresponding counts for each document . The term vectors for a document collection can be organized into a term by document co-occurrence matrix . When directly using these representations , synonyms and polysemous terms , that is , terms with multiple senses or meanings , are not handled well . Methods for smoothing the term distributions through the use of latent classes have been shown to improve the performance of a number of information access tasks , including retrieval over smaller collections ( Deerwester et al . , 1990 ) , text segmentation ( Brants et al . , 2002 ) , and text classification ( Wu and Gunopulos , 2002 ) . The Probabilistic Latent Semantic Analysis model ( PLSA ) ( Hofmann , 1999 ) provides a probabilistic framework that attempts to capture polysemy and synonymy in text for applications such as retrieval and segmentation . It uses a mixture decomposition to model the co-occurrence data , and the probabilities of words and documents are obtained by a convex combination of the aspects . The mixture approximation has a well defined probability distribution and the factors have a clear probabilistic meaning in terms of the mixture component distributions . The PLSA model computes the relevant probability distributions by selecting the model parameter values that maximize the probability of the observed data , i.e. , the likelihood function . The standard method for maximum likelihood estimation is the Expectation Maximization ( EM ) algorithm . For a given initialization , the likelihood function increases with EM iterations until a local maximum is reached , rather than a global maximum , so that the quality of the solution depends on the initialization of the model . Additionally , the likelihood values across different initializations are not comparable , as we will show . Thus , the likelihood function computed over the training data can not be used as a predictor of model performance across different models . Rather than trying to predict the best performing model from a set of models , in this paper we focus on finding a good way to initialize the PLSA model . We will present a framework for using Latent Semantic Analysis ( LSA ) ( Deerwester et al . , 1990 ) to better initialize the parameters of a corresponding PLSA model . The EM algorithm is then used to further refine the initial estimate . This combination of LSA and PLSA leverages the advantages of both . This paper is organized as follows : in section 2 , we review related work in the area . In section 3 , we summarize related work on LSA and its probabilistic interpretation . In section 4 we review the PLSA model and in section 5 we present our method for initializing a PLSA model using LSA model parameters . In section 6 , we evaluate the performance of our framework on a text segmentation task and several smaller information retrieval tasks . And in section 7 , we summarize our results and give directions for future work . We have presented LSA-PLSA , an approach for improving the performance of PLSA by leveraging the best features of PLSA and LSA . Our approach uses LSA to initialize a PLSA model , allowing for arbitrary weighting schemes to be incorporated into a PLSA model while leveraging the optimization used to improve the estimate of the PLSA parameters . We have evaluated the proposed framework on two tasks : personalsize information retrieval and text segmentation . The LSA-PLSA model outperformed PLSA on all tasks . And in all cases , combining PLSA-based models outperformed a single model . The best performance was obtained with combined models when one of the models was the LSA-PLSA model . When combining multiple PLSA models , the use of LSA-PLSA in combination with either two PLSA models or one PLSA and one LSA model improved performance while reducing the running time over the combination of four or more PLSA models as used by others . Future areas of investigation include quantifying the expected performance of the LSAinitialized PLSA model by comparing performance to that of the empirically best performing model and examining whether tempered EM could further improve performance .
EM algorithm-baed Probabilistic latent semantic analysis models provide high variance in performance and models with different initializations are not comparable.
They propose to use Latent Semantic Analysis to initialize probabilistic latent semantic analysis models, EM algorithm is further used to refine the initial estimate.
They show that the model initialized in the proposed method always outperforms existing methods.