Machine learning approaches to multi-label document classification have to date largely relied on discriminative modeling techniques such as support vector machines. A drawback of these approaches is that performance rapidly drops off as the total number of labels and the number of labels per document increase. This problem is amplified when the label frequencies exhibit the type of highly skewed distributions that are often observed in real-world datasets. In this paper we investigate a class of generative statistical topic models for multi-label documents that associate individual word tokens with different labels. We investigate the advantages of this approach relative to discriminative models, particularly with respect to classification problems involving large numbers of relatively rare labels. We compare the performance of generative and discriminative approaches on document labeling tasks ranging from datasets with several thousand labels to datasets with tens of labels. The experimental results indicate that probabilistic generative models can achieve competitive multi-label classification performance compared to discriminative methods, and have advantages for datasets with many labels and skewed label frequencies.
More recently @cite_7 demonstrated that the probabilistic framework of conditional random fields showed promise for multilabel classification, compared to discriminative classifiers, as the number of labels within test documents increased. In follow-up work on these models, @cite_6 illustrated that this approach has the further benefit of being able to naturally incorporate unlabeled data for semi-supervised learning. A drawback of the CRF approach is scalability, particularly when accounting for label dependencies. Exact inference is tractable only for about 3-12 [labels]'' . Alternatives to exact inference considered in include a supported inference" method which learns only to classify the label combinations that occur in the training set, and a binary-pruning method that employs an intelligent pruning method which ignores dependencies between all but the most commonly observed pairs of labels. Although this method may improve upon approaches that ignore dependencies when restricted to datasets with few labels and many examples (such as traditional benchmark datasets), it seems unlikely that any such methods will be able to properly account for dependencies in datasets with power-law frequency statistics (since nearly all dependencies in these datasets are between labels which have very sparse training data).
@cite_6 @cite_7
2101101940 1976526581
Supervised topic models utilize document's side information for discovering predictive low dimensional representations of documents; and existing models apply likelihood-based estimation. In this paper, we present a max-margin supervised topic model for both continuous and categorical response variables. Our approach, the maximum entropy discrimination latent Dirichlet allocation (MedLDA), utilizes the max-margin principle to train supervised topic models and estimate predictive topic representations that are arguably more suitable for prediction. We develop efficient variational methods for posterior inference and demonstrate qualitatively and quantitatively the advantages of MedLDA over likelihood-based topic models on movie review and 20 Newsgroups data sets. Receiver Operator Characteristic (ROC) curves are commonly used to present results for binary decision problems in machine learning. However, when dealing with highly skewed datasets, Precision-Recall (PR) curves give a more informative picture of an algorithm's performance. We show that a deep connection exists between ROC space and PR space, such that a curve dominates in ROC space if and only if it dominates in PR space. A corollary is the notion of an achievable PR curve, which has properties much like the convex hull in ROC space; we show an efficient algorithm for computing this curve. Finally, we also note differences in the two types of curves are significant for algorithm design. For example, in PR space it is incorrect to linearly interpolate between points. Furthermore, algorithms that optimize the area under the ROC curve are not guaranteed to optimize the area under the PR curve.