Topic models have been extensively used to organize and interpret the contents of large, unstructured corpora of text documents. Although topic models often perform well on traditional training vs. test set evaluations, it is often the case that the results of a topic model do not align with human interpretation. This interpretability fallacy is largely due to the unsupervised nature of topic models, which prohibits any user guidance on the results of a model. In this paper, we introduce a semi-supervised method called topic supervised non-negative matrix factorization (TS-NMF) that enables the user to provide labeled example documents to promote the discovery of more meaningful semantic structure of a corpus. In this way, the results of TS-NMF better match the intuition and desired labeling of the user. The core of TS-NMF relies on solving a non-convex optimization problem for which we derive an iterative algorithm that is shown to be monotonic and convergent to a local optimum. We demonstrate the practical utility of TS-NMF on the Reuters and PubMed corpora, and find that TS-NMF is especially useful for conceptual or broad topics, where topic key terms are not well understood. Although identifying an optimal latent structure for the data is not a primary objective of the proposed approach, we find that TS-NMF achieves higher weighted Jaccard similarity scores than the contemporary methods, (unsupervised) NMF and latent Dirichlet allocation, at supervision rates as low as 10 to 20 .
A corpus of @math documents and @math terms can be represented by a @math term-document matrix @math containing non-negative entries @math that quantify the importance of term @math in document @math . The choice of weights @math is dependent upon the application, but is typically calculated using some function of term frequency (TF) and inverse document frequency (IDF) (see @cite_12 for a review). Mathematically, topic models are mappings of @math to a lower dimensional representation of @math involving the @math topics describing the documents. Existing topic modeling approaches generally fall into two classes of methods: matrix decomposition methods, which seek a low dimensional representation of @math through a factorization into two or more low-rank matrices, and probabilistic topic modeling methods, which seek a generative statistical model for @math . Here, we describe each class of methods in more detail, paying special attention to the works that are most closely related to TS-NMF.
@cite_12
2128521126 2952389066 2058846446 2528414367
Topic Modeling is an approach used for automatic comprehension and classification of data in a variety of settings, and perhaps the canonical application is in uncovering thematic structure in a corpus of documents. A number of foundational works both in machine learning and in theory have suggested a probabilistic model for documents, whereby documents arise as a convex combination of (i.e. distribution on) a small number of topic vectors, each topic vector being a distribution on words (i.e. a vector of word-frequencies). Similar models have since been used in a variety of application areas, the Latent Dirichlet Allocation or LDA model of is especially popular. Theoretical studies of topic modeling focus on learning the model's parameters assuming the data is actually generated from it. Existing approaches for the most part rely on Singular Value Decomposition (SVD), and consequently have one of two limitations: these works need to either assume that each document contains only one topic, or else can only recover the span of the topic vectors instead of the topic vectors themselves. This paper formally justifies Nonnegative Matrix Factorization (NMF) as a main tool in this context, which is an analog of SVD where all vectors are nonnegative. Using this tool we give the first polynomial-time algorithm for learning topic models without the above two limitations. The algorithm uses a fairly mild assumption about the underlying topic matrix called separability, which is usually found to hold in real-life data. Perhaps the most attractive feature of our algorithm is that it generalizes to yet more realistic models that incorporate topic-topic correlations, such as the Correlated Topic Model (CTM) and the Pachinko Allocation Model (PAM). We hope that this paper will motivate further theoretical results that use NMF as a replacement for SVD -- just as NMF has come to replace SVD in many applications. Topic Modeling is an approach used for automatic comprehension and classification of data in a variety of settings, and perhaps the canonical application is in uncovering thematic structure in a corpus of documents. A number of foundational works both in machine learning and in theory have suggested a probabilistic model for documents, whereby documents arise as a convex combination of (i.e. distribution on) a small number of topic vectors, each topic vector being a distribution on words (i.e. a vector of word-frequencies). Similar models have since been used in a variety of application areas; the Latent Dirichlet Allocation or LDA model of is especially popular. Theoretical studies of topic modeling focus on learning the model's parameters assuming the data is actually generated from it. Existing approaches for the most part rely on Singular Value Decomposition(SVD), and consequently have one of two limitations: these works need to either assume that each document contains only one topic, or else can only recover the span of the topic vectors instead of the topic vectors themselves. This paper formally justifies Nonnegative Matrix Factorization(NMF) as a main tool in this context, which is an analog of SVD where all vectors are nonnegative. Using this tool we give the first polynomial-time algorithm for learning topic models without the above two limitations. The algorithm uses a fairly mild assumption about the underlying topic matrix called separability, which is usually found to hold in real-life data. A compelling feature of our algorithm is that it generalizes to models that incorporate topic-topic correlations, such as the Correlated Topic Model and the Pachinko Allocation Model. We hope that this paper will motivate further theoretical results that use NMF as a replacement for SVD - just as NMF has come to replace SVD in many applications. Non-negative matrix factorization (NMF) has been successfully applied in document clustering. However, experiments on short texts, such as microblogs, Q&A documents and news titles, suggest unsatisfactory performance of NMF. An major reason is that the traditional term weighting schemes, like binary weight and tfidf, cannot well capture the terms' discriminative power and importance in short texts, due to the sparsity of data. To tackle this problem, we proposed a novel term weighting scheme for NMF, derived from the Normalized Cut (Ncut) problem on the term affinity graph. Different from idf, which emphasizes discriminability on document level, the Ncut weighting measures terms' discriminability on term level. Experiments on two data sets show our weighting scheme significantly boosts NMF's performance on short text clustering. Nonnegative matrix factorization (NMF) has beenwidely applied in many domains. In document analysis, it hasbeen increasingly used in topic modeling applications, where aset of underlying topics are revealed by a low-rank factor matrixfrom NMF. However, it is often the case that the resulting topicsgive only general topic information in the data, which tends notto convey much information. To tackle this problem, we proposea novel ensemble model of nonnegative matrix factorizationfor discovering high-quality local topics. Our method leveragesthe idea of an ensemble model, which has been successfulin supervised learning, into an unsupervised topic modelingcontext. That is, our model successively performs NMF givena residual matrix obtained from previous stages and generatesa sequence of topic sets. Our algorithm for updating the inputmatrix has novelty in two aspects. The first lies in utilizing theresidual matrix inspired by a state-of-the-art gradient boostingmodel, and the second stems from applying a sophisticatedlocal weighting scheme on the given matrix to enhance thelocality of topics, which in turn delivers high-quality, focusedtopics of interest to users. We evaluate our proposed method bycomparing it against other topic modeling methods, such as afew variants of NMF and latent Dirichlet allocation, in termsof various evaluation measures representing topic coherence, diversity, coverage, computing time, and so on. We also presentqualitative evaluation on the topics discovered by our methodusing several real-world data sets.