Recently by the development of the Internet and the Web, different types of social media such as web blogs become an immense source of text data. Through the processing of these data, it is possible to discover practical information about different topics, individual’s opinions and a thorough understanding of the society. Therefore, applying models which can automatically extract the subjective information from documents would be efficient and helpful. Topic modeling methods and sentiment analysis are the raised topics in natural language processing and text mining fields. In this paper a new structure for joint sentiment-topic modeling based on a Restricted Boltzmann Machine (RBM) which is a type of neural networks is proposed. By modifying the structure of RBM as well as appending a layer which is analogous to sentiment of text data to it, we propose a generative structure for joint sentiment topic modeling based on neural networks. The proposed method is supervised and trained by the Contrastive Divergence algorithm. The new attached layer in the proposed model is a layer with the multinomial probability distribution which can be used in text data sentiment classification or any other supervised application. The proposed model is compared with existing models in the experiments such as evaluating as a generative model, sentiment classification, information retrieval and the corresponding results demonstrate the efficiency of the method.
The Restricted Boltzmann Machine (RBM) is an unsupervised two-layer neural network for the estimation of distribution of binary input data. This generative probabilistic model was first introduced in 1986 by Smolensky @cite_3 and was later developed by Hinton in 2002 @cite_9 . Inspired by the RBM model, in 2011 introduced the Neural Autoregressive Distribution Estimation (NADE) @cite_12 , which is an unsupervised generative probabilistic method for modeling the probability of discrete data. NADE eliminated the limitation of RBM in high dimensional joint probability estimations by the use of fully visible Bayes networks for probability calculations. The earliest neural network-based topic model is Replicated Softmax Model (RS) introduced by Hinton and Salakhutdinov in 2009 @cite_6 , which is an extension of the RBM model used to detect the distribution of topics in text data @cite_6 . In 2012, Larochelle and Lauly combined NADE and RS to develop an unsupervised neural network-based topic modeling method called the Document Neural Autoregressive Distribution Estimation (DocNADE) @cite_0 .
@cite_9 @cite_3 @cite_6 @cite_0 @cite_12
2551541394 2145195026 2293139688 2018168021
Restricted Boltzmann Machine (RBM) is an important generative model modeling vectorial data. While applying an RBM in practice to images, the data have to be vectorized. This results in high-dimensional data and valuable spatial information has got lost in vectorization. In this paper, a Matrix-Variate Restricted Boltzmann Machine (MVRBM) model is proposed by generalizing the classic RBM to explicitly model matrix data. In the new RBM model, both input and hidden variables are in matrix forms which are connected by bilinear transforms. The MVRBM has much less model parameters while retaining comparable performance as the classic RBM. The advantages of the MVRBM have been demonstrated on three real-world applications: handwritten digit denoising, reconstruction and recognition. The recurrent temporal restricted Boltzmann machine (RTRBM) is a probabilistic time-series model. The topology of the RTRBM graphical model, however, assumes full connectivity between all the pairs of visible units and hidden units, thereby ignoring the dependency structure within the observations. Learning this structure has the potential for not only improving the prediction performance, but also revealing important dependency patterns in the data. For example, given a meteorological dataset, we could identify regional weather patterns. In this work, we propose a new class of RTRBM, which we refer to as the structured RTRBM (SRTRBM), which explicitly uses a graph to model the dependency structure. Our technique is related to methods such as graphical lasso, which are used to learn the topology of Gaussian graphical models. We also develop a spike-and-slab version of the RTRBM, and combine it with the SRTRBM to learn dependency structures in datasets with real-valued observations. Our experimental results using synthetic and real datasets demonstrate that the SRTRBM can significantly improve the prediction performance of the RTRBM, particularly when the number of visible units is large and the size of the training set is small. It also reveals the dependency structures underlying our benchmark datasets. Restricted Boltzmann Machines (RBMs) are an important class of latent variable models for representing vector data. An under-explored area is multimode data, where each data point is a matrix or a tensor. Standard RBMs applying to such data would require vectorizing matrices and tensors, thus resulting in unnecessarily high dimensionality and at the same time, destroying the inherent higher-order interaction structures. This paper introduces Tensor-variate Restricted Boltzmann Machines (TvRBMs) which generalize RBMs to capture the multiplicative interaction between data modes and the latent variables. TvRBMs are highly compact in that the number of free parameters grows only linear with the number of modes. We demonstrate the capacity of TvRBMs on three real-world applications: handwritten digit classification, face recognition and EEG-based alcoholic diagnosis. The learnt features of the model are more discriminative than the rivals, resulting in better classification performance. Restricted Boltzmann machines (RBMs) are probabilistic graphical models that can be interpreted as stochastic neural networks. They have attracted much attention as building blocks for the multi-layer learning systems called deep belief networks, and variants and extensions of RBMs have found application in a wide range of pattern recognition tasks. This tutorial introduces RBMs from the viewpoint of Markov random fields, starting with the required concepts of undirected graphical models. Different learning algorithms for RBMs, including contrastive divergence learning and parallel tempering, are discussed. As sampling from RBMs, and therefore also most of their learning algorithms, are based on Markov chain Monte Carlo (MCMC) methods, an introduction to Markov chains and MCMC techniques is provided. Experiments demonstrate relevant aspects of RBM training. HighlightsWe review the state-of-the-art in training restricted Boltzmann machines (RBMs) from the perspective of graphical models.Variants and extensions of RBMs are used in a wide range of pattern recognition tasks.The required background on graphical models and Markov chain Monte Carlo methods is provided.Theoretical and experimental results are presented.