Representation Learning: A Review and New Perspectives <s> Pseudolikelihood, Ratio Matching and More <s> In rather formal terms, the situation with which this paper is concerned may be described as follows. We are given a fixed system of n sites, labelled by the first n positive integers, and an associated vector x of observations, Xi, . . ., Xn, which, in turn, is presumed to be a realization of a vector X of (dependent) random variables, Xi, . . ., X.. In practice, the sites may represent points or regions in space and the random variables may be either continuous or discrete. The main statistical objectives are the following: firstly, to provide a means of using the available concomitant information, particularly the configuration of the sites, to attach a plausible probability distribution to the random vector X; secondly, to estimate any unknown parameters in the distribution from the realization x; thirdly, where possible, to quantify the extent of disagreement between hypothesis and observation. <s> BIB001 </s> Representation Learning: A Review and New Perspectives <s> Pseudolikelihood, Ratio Matching and More <s> Many probabilistic models are only defined up to a normalization constant. This makes maximum likelihood estimation of the model parameters very difficult. Typically, one then has to resort to Markov Chain Monte Carlo methods, or approximations of the normalization constant. Previously, a method called score matching was proposed for computationally efficient yet (locally) consistent estimation of such models. The basic form of score matching is valid, however, only for models which define a differentiable probability density function over R^n. Therefore, some extensions of the framework are proposed. First, a related method for binary variables is proposed. Second, it is shown how to estimate non-normalized models defined in the non-negative real domain, i.e. R"+^n. As a further result, it is shown that the score matching estimator can be obtained in closed form for some exponential families. <s> BIB002 </s> Representation Learning: A Review and New Perspectives <s> Pseudolikelihood, Ratio Matching and More <s> We describe a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense (grammatically and semantically) using a language model. The entire network is trained jointly on all these tasks using weight-sharing, an instance of multitask learning. All the tasks use labeled data except the language model which is learnt from unlabeled text and represents a novel form of semi-supervised learning for the shared tasks. We show how both multitask learning and semi-supervised learning improve the generalization of the shared tasks, resulting in state-of-the-art-performance. <s> BIB003 </s> Representation Learning: A Review and New Perspectives <s> Pseudolikelihood, Ratio Matching and More <s> Recent research has seen the proposal of several new inductive principles designed specifically to avoid the problems associated with maximum likelihood learning in models with intractable partition functions. In this paper, we study learning methods for binary restricted Boltzmann machines (RBMs) based on ratio matching and generalized score matching. We compare these new RBM learning methods to a range of existing learning methods including stochastic maximum likelihood, contrastive divergence, and pseudo-likelihood. We perform an extensive empirical evaluation across multiple tasks and data sets. <s> BIB004 </s> Representation Learning: A Review and New Perspectives <s> Pseudolikelihood, Ratio Matching and More <s> We present a new estimation principle for parameterized statistical models. The idea is to perform nonlinear logistic regression to discriminate between the observed data and some artificially generated noise, using the model log-density function in the regression nonlinearity. We show that this leads to a consistent (convergent) estimator of the parameters, and analyze the asymptotic variance. In particular, the method is shown to directly work for unnormalized models, i.e. models where the density function does not integrate to one. The normalization constant can be estimated just like any other parameter. For a tractable ICA model, we compare the method with other estimation methods that can be used to learn unnormalized models, including score matching, contrastive divergence, and maximum-likelihood where the normalization constant is estimated with importance sampling. Simulations show that noise-contrastive estimation offers the best trade-off between computational and statistical efficiency. The method is then applied to the modeling of natural images: We show that the method can successfully estimate a large-scale two-layer model and a Markov random field. <s> BIB005 </s> Representation Learning: A Review and New Perspectives <s> Pseudolikelihood, Ratio Matching and More <s> Image annotation datasets are becoming larger and larger, with tens of millions of images and tens of thousands of possible annotations. We propose a strongly performing method that scales to such datasets by simultaneously learning to optimize precision at k of the ranked list of annotations for a given image and learning a low-dimensional joint embedding space for both images and annotations. Our method both outperforms several baseline methods and, in comparison to them, is faster and consumes less memory. We also demonstrate how our method learns an interpretable model, where annotations with alternate spellings or even languages are close in the embedding space. Hence, even when our model does not predict the exact annotation given by a human labeler, it often predicts similar annotations, a fact that we try to quantify by measuring the newly introduced "sibling" precision metric, where our method also obtains excellent results. <s> BIB006 </s> Representation Learning: A Review and New Perspectives <s> Pseudolikelihood, Ratio Matching and More <s> Denoising autoencoders have been previously shown to be competitive alternatives to restricted Boltzmann machines for unsupervised pretraining of each layer of a deep architecture. We show that a simple denoising autoencoder training criterion is equivalent to matching the score (with respect to the data) of a specific energy-based model to that of a nonparametric Parzen density estimator of the data. This yields several useful insights. It defines a proper probabilistic model for the denoising autoencoder technique, which makes it in principle possible to sample from them or rank examples by their energy. It suggests a different way to apply score matching that is related to learning to denoise and does not require computing second derivatives. It justifies the use of tied weights between the encoder and decoder and suggests ways to extend the success of denoising autoencoders to a larger family of energy-based models. <s> BIB007 </s> Representation Learning: A Review and New Perspectives <s> Pseudolikelihood, Ratio Matching and More <s> Standard maximum likelihood estimation cannot be applied to discrete energy-based models in the general case because the computation of exact model probabilities is intractable. Recent research has seen the proposal of several new estimators designed specifically to overcome this intractability, but virtually nothing is known about their theoretical properties. In this paper, we present a generalized estimator that unifies many of the classical and recently proposed estimators. We use results from the standard asymptotic theory for M-estimators to derive a generic expression for the asymptotic covariance matrix of our generalized estimator. We apply these results to study the relative statistical efficiency of classical pseudolikelihood and the recently-proposed ratio matching estimator. <s> BIB008 </s> Representation Learning: A Review and New Perspectives <s> Pseudolikelihood, Ratio Matching and More <s> Open-text semantic parsers are designed to interpret any statement in natural language by inferring a corresponding meaning representation (MR – a formal representation of its sense). Unfortunately, large scale systems cannot be easily machine-learned due to a lack of directly supervised data. We propose a method that learns to assign MRs to a wide range of text (using a dictionary of more than 70,000 words mapped to more than 40,000 entities) thanks to a training scheme that combines learning from knowledge bases (e.g. WordNet) with learning from raw text. The model jointly learns representations of words, entities and MRs via a multi-task training process operating on these diverse sources of data. Hence, the system ends up providing methods for knowledge acquisition and wordsense disambiguation within the context of semantic parsing in a single elegant framework. Experiments on these various tasks indicate the promise of the approach. <s> BIB009
While CD, SML, and FPCD are by far the most popular methods for training RBMs and RBM-based models, all of these methods are perhaps most naturally described as offering different approximations to maximum likelihood training. There exist other inductive principles that are alternatives to maximum likelihood that can also be used to train RBMs. In particular, these include pseudolikelihood BIB001 and ratio matching BIB002 . Both of these inductive principles attempt to avoid explicitly dealing with the partition function, and their asymptotic efficiency has been analyzed BIB008 . Pseudolikelihood seeks to maximize the product of all 1D conditional distributions of the form P ðx d j x nd Þ, while ratio matching can be interpreted as an extension of score matching  to discrete data types. Both methods amount to weighted differences of the gradient of the RBM free energy 13 evaluated at a data point and at neighboring points. One potential drawback of these methods is that depending on the parameterization of the energy function, their computational requirements may scale up to Oðn d Þ worse than CD, SML, FPCD, or denoising score matching  , BIB007 , discussed below. Marlin et al. BIB004 empirically compared all of these methods (except denoising score matching) on a range of classification, reconstruction, and density modeling tasks and found that, in general, SML provided the best combination of overall performance and computational tractability. However, in a later study, the same authors  found denoising score matching to be a competitive inductive principle both in terms of classification performance (with respect to SML) and in terms of computational efficiency (with respect to analytically obtained score matching). Denoising score matching is a special case of the denoising autoencoder (DAE) training criterion (Section 7.2.2) when the reconstruction error residual equals a gradient, i.e., the score function associated with an energy function, as shown in BIB007 . In the spirit of the Boltzmann machine gradient (15), several approaches have been proposed to train energybased models. One is noise-contrastive estimation BIB005 , in which the training criterion is transformed into a probabilistic classification problem: BIB003 , BIB009 , BIB006 .