A Survey on Context Learning <s> Restricted Boltzmann Machine Based Analysis <s> Berners-Lee's compelling vision of a Semantic Web is hindered by a chicken-and-egg problem, which can be best solved by a bootstrapping method - creating enough structured data to motivate the development of applications. This paper argues that autonomously "Semantifying Wikipedia" is the best way to solve the problem. We choose Wikipedia as an initial data source, because it is comprehensive, not too large, high-quality, and contains enough manually-derived structure to bootstrap an autonomous, self-supervised process. We identify several types of structures which can be automatically enhanced in Wikipedia (e.g., link structure, taxonomic data, infoboxes, etc.), and we describea prototype implementation of a self-supervised, machine learning system which realizes our vision. Preliminary experiments demonstrate the high precision of our system's extracted data - in one case equaling that of humans. <s> BIB001 </s> A Survey on Context Learning <s> Restricted Boltzmann Machine Based Analysis <s> We describe a new family of non-linear sequence models that are substantially more powerful than hidden Markov models or linear dynamical systems. Our models have simple approximate inference and learning procedures that work well in practice. Multilevel representations of sequential data can be learned one hidden layer at a time, and adding extra hidden layers improves the resulting generative models. The models can be trained with very high-dimensional, very non-linear data such as raw pixel sequences. Their performance is demonstrated using synthetic video sequences of two balls bouncing in a box. <s> BIB002 </s> A Survey on Context Learning <s> Restricted Boltzmann Machine Based Analysis <s> The Temporal Restricted Boltzmann Machine (TRBM) is a probabilistic model for sequences that is able to successfully model (i.e., generate nice-looking samples of) several very high dimensional sequences, such as motion capture data and the pixels of low resolution videos of balls bouncing in a box. The major disadvantage of the TRBM is that exact inference is extremely hard, since even computing a Gibbs update for a single variable of the posterior is exponentially expensive. This difficulty has necessitated the use of a heuristic inference procedure, that nonetheless was accurate enough for successful learning. In this paper we introduce the Recurrent TRBM, which is a very slight modification of the TRBM for which exact inference is very easy and exact gradient learning is almost tractable. We demonstrate that the RTRBM is better than an analogous TRBM at generating motion capture and videos of bouncing balls. <s> BIB003 </s> A Survey on Context Learning <s> Restricted Boltzmann Machine Based Analysis <s> We introduce a two-layer undirected graphical model, called a "Replicated Softmax", that can be used to model and automatically extract low-dimensional latent semantic representations from a large unstructured collection of documents. We present efficient learning and inference algorithms for this model, and show how a Monte-Carlo based method, Annealed Importance Sampling, can be used to produce an accurate estimate of the log-probability the model assigns to test data. This allows us to demonstrate that the proposed model is able to generalize much better compared to Latent Dirichlet Allocation in terms of both the log-probability of held-out documents and the retrieval accuracy. <s> BIB004 </s> A Survey on Context Learning <s> Restricted Boltzmann Machine Based Analysis <s> We present a new learning algorithm for Boltzmann machines that contain many layers of hidden variables. Data-dependent expectations are estimated using a variational approximation that tends to focus on a single mode, and dataindependent expectations are approximated using persistent Markov chains. The use of two quite different techniques for estimating the two types of expectation that enter into the gradient of the log-likelihood makes it practical to learn Boltzmann machines with multiple hidden layers and millions of parameters. The learning can be made more efficient by using a layer-by-layer “pre-training” phase that allows variational inference to be initialized with a single bottomup pass. We present results on the MNIST and NORB datasets showing that deep Boltzmann machines learn good generative models and perform well on handwritten digit and visual object recognition tasks. <s> BIB005 </s> A Survey on Context Learning <s> Restricted Boltzmann Machine Based Analysis <s> We introduce a Deep Boltzmann Machine model suitable for modeling and extracting latent semantic representations from a large unstructured collection of documents. We overcome the apparent difficulty of training a DBM with judicious parameter tying. This parameter tying enables an efficient pretraining algorithm and a state initialization scheme that aids inference. The model can be trained just as efficiently as a standard Restricted Boltzmann Machine. Our experiments show that the model assigns better log probability to unseen data than the Replicated Softmax model. Features extracted from our model outperform LDA, Replicated Softmax, and DocNADE models on document retrieval and document classification tasks. <s> BIB006
Technically, a restricted Boltzmann machine is a generative stochastic neural network that can learn a probability distribution over the set of inputs. A Boltzmann machine is a symmetric network of binary units. The binary units are composed of a layer of visible units v 2 f0; 1g D and a layer of hidden units h 2 f0; 1g P : Accordingly, when a Boltzmann machine is in the state of fv; hg, its energy is defined as where fL; J; W; b; cg are the model parameters: L; J; W denote the symmetric interaction weights between visible units and visible units, hidden and hidden units, visible and hidden units respectively, and b and c are the bias vectors, and v 0 denotes v transpose to avoid confusion. By setting L ¼ 0 and J ¼ 0, we obtain the well-known RBM which has no visibleto-visible and hidden-to-hidden interactions. The probability distribution on state fv; hg is defined based on its energy pðv; hÞ ¼ 1 P v P h expðÀEðv; hÞÞ expðÀEðv; hÞÞ: As the units are binary, this yields the conditional distributions over visible units and hidden units where sðxÞ ¼ 1=ð1 þ e Àx Þ is the sigmoid logistic function, which enables efficient Gibbs sampling for one layer given the other layer. Based on the observation of the visible units v, the log likelihood L to maximize is L ¼ log pðvÞ ¼ log P h pðv; hÞ. Using gradient ascent to reach the maximum likelihood in this model is intractable as there are exponentially many items involved in calculating the normalizing constant in Equation BIB001 . To obtain an efficient approximation, a Monte Carlo estimation, i.e., the Contrastive Divergence method  is usually adopted. The Replicated Softmax model is an RBM based language model that is used to extract low dimension semantic representations from a collection of documents by analyzing the context information in each document BIB004 . As shown in Fig. 8 , each visible unit denotes one input word in a document or a context, and the hidden units denote the latent semantic representations. The proposed model adopts one-hot representation for words. Suppose the vocabulary size is K, if we denote e k the one-hot vector with its kth component set to 1, then the conditional distribution of the ith word becomes Usually the document contexts are of different length, which results in different amounts of the visible units and weight parameters. The Replicated Softmax addressed this issue by sharing the weights among different visible units connected to the same hidden unit. In order to obtain some degree of flexibility in defining the prior over the hidden units and extract deeper latent semantics, deep Boltzmann machines (DBM) BIB005 are adopted as the basic model. The Over-Replicated Softmax is an RBM based language model with two hidden layers BIB006 . To avoid the expensive training and inference of DBM, the top layer, i.e., the second hidden layer is designed to be shared softmax units as the visible layer. Sometimes, context information could be not only spatial, but also temporal. This results in effective adaptations: the Temporal Restricted Boltzmann Machine (TRBM) BIB002 . As a probabilistic model, the TRBM is a directed graphical model consisting of a sequence of RBMs, where the state of historical RBMs determine the biases of the RBM in next timestamp. The probability distribution on state v t ; h t at time t is where Z is the normalizing constant, m is the temporal parameter determining how many historical states are considered, v t and h t denote the states of the visible and hidden layer at time t. A 0 is the weights between the current visible and hidden layer, corresponding to weights W in the standard RBM. The notation v tÀ1 tÀm and h tÀ1 tÀm stand for the historical states of the visible and hidden layer from time t À m to time t À 1, i.e., fv tÀ1 ; . . . ; v tÀm g and fh tÀ1 ; . . . ; h tÀm g. Fig. 9 illustrates how the TRBM model denoises sequences by learning the temporal context information: the top row is an image sequence; the second row is the same image sequence corrupted by noise that is highly correlated in space and time; the third row is the image sequence denoised by a single-hidden-layer TRBM and the bottom row is the image sequence denoised by a two-hidden-layer TRBM. Later on, a similar extension based on Recurrent Neural Network is proposed and named Recurrent Temporal Restricted Boltzmann Machine (RTRBM) BIB003 .