Representing texts as fixed-length vectors is central to many language processing tasks. Most traditional methods build text representations based on the simple Bag-of-Words (BoW) representation, which loses the rich semantic relations between words. Recent advances in natural language processing have shown that semantically meaningful representations of words can be efficiently acquired by distributed models, making it possible to build text representations based on a better foundation called the Bag-of-Word-Embedding (BoWE) representation. However, existing text representation methods using BoWE often lack sound probabilistic foundations or cannot well capture the semantic relatedness encoded in word vectors. To address these problems, we introduce the Spherical Paragraph Model (SPM), a probabilistic generative model based on BoWE, for text representation. SPM has good probabilistic interpretability and can fully leverage the rich semantics of words, the word co-occurrence information as well as the corpus-wide information to help the representation learning of texts. Experimental results on topical classification and sentiment analysis demonstrate that SPM can achieve new state-of-the-art performances on several benchmark datasets.
The vMF distribution has properties analogous to those of the multi-variate Gaussian distribution for data in @math , parameterized by cosine similarity rather than Euclidean distance. Evidence suggests that this type of directional measure (, cosine similarity) is often superior to Euclidean distance in high dimensions @cite_1 @cite_9 .
@cite_9 @cite_1
2123504323 1617750321 2163952039 2105750764
High dimensional directional data is becoming increasingly important in contemporary applications such as analysis of text and gene-expression data. A natural model for multi-variate directional data is provided by the von Mises-Fisher (vMF) distribution on the unit hypersphere that is analogous to the multi-variate Gaussian distribution in Rd. In this paper, we propose modeling complex directional data as a mixture of vMF distributions. We derive and analyze two variants of the Expectation Maximization (EM) framework for estimating the parameters of this mixture. We also propose two clustering algorithms corresponding to these variants. An interesting aspect of our methodology is that the spherical kmeans algorithm (kmeans with cosine similarity) can be shown to be a special case of both our algorithms. Thus, modeling text data by vMF distributions lends theoretical validity to the use of cosine similarity which has been widely used by the information retrieval community. As part of experimental validation, we present results on modeling high-dimensional text and gene-expression data as a mixture of vMF distributions. The results indicate that our approach yields superior clusterings especially for difficult clustering tasks in high-dimensional spaces. Von Mises-Fisher (vMF) Distribution is one of the most commonly used distributions for fitting directional data. Mixtures of vMF (MovMF) distributions have been used successfully in many applications. One of the important problems in mixture models is the problem of local minima of the objective function. Therefore, approaches to avoid local minima problem is essential in improving the performance. Recently, an algorithm called k-means++ was introduced in the literature and used successfully for finding initial parameters for mixtures of Gaussian (MoG) distributions. In this paper, we adopt this algorithm for finding good initializations for MovMF distributions. We show that MovMF distribution will lead to the same cost function as MoGs and therefore similar guarantee as the case of MoG distributions will also hold here. We also demonstrate the performance of the method on some real datasets. Instance-based learning techniques typically handle continuous and linear input values well, but often do not handle nominal input attributes appropriately. The Value Difference Metric (VDM) was designed to find reasonable distance values between nominal attribute values, but it largely ignores continuous attributes, requiring discretization to map continuous values into nominal values. This paper proposes three new heterogeneous distance functions, called the Heterogeneous Value Difference Metric (HVDM), the Interpolated Value Difference Metric (IVDM), and the Windowed Value Difference Metric (WVDM). These new distance functions are designed to handle applications with nominal attributes, continuous attributes, or both. In experiments on 48 applications the new distance metrics achieve higher classification accuracy on average than three previous distance functions on those datasets that have both nominal and continuous attributes. High angular resolution diffusion imaging (HARDI) permits the computation of water molecule displacement probabilities over the sphere. This probability is often referred to as the orientation distribution function (ODF). In this paper we present a novel model for representing this diffusion ODF namely, a mixture of von Mises-Fisher (vMF) distributions. Our model is compact in that it requires very few parameters to represent complicated ODF geometries which occur specifically in the presence of heterogeneous nerve fiber orientations. We present a Riemannian geometric framework for computing intrinsic distances (in closed-form) and for performing interpolation between ODFs represented by vMF mixtures. We also present closed-form equations for entropy and variance based anisotropy measures that are then computed and illustrated for real HARDI data from a rat brain.