Representation Learning: A Review and New Perspectives <s> Evaluating and Monitoring Performance <s> Markov Random Fields (MRFs) have proven very powerful both as density estimators and feature extractors for classification. However, their use is often limited by an inability to estimate the partition function Z. In this paper, we exploit the gradient descent training procedure of restricted Boltzmann machines (a type of MRF) to track the log partition function during learning. Our method relies on two distinct sources of information: (1) estimating the change AZ incurred by each gradient update, (2) estimating the difference in Z over a small set of tempered distributions using bridge sampling. The two sources of information are then combined using an inference procedure similar to Kalman filtering. Learning MRFs through Tempered Stochastic Maximum Likelihood, we can estimate Z using no more temperatures than are required for learning. Comparing to both exact values and estimates using annealed importance sampling (AIS), we show on several datasets that our method is able to accurately track the log partition function. In contrast to AIS, our method provides this estimate at each time-step, at a computational cost similar to that required for training alone. <s> BIB001 </s> Representation Learning: A Review and New Perspectives <s> Evaluating and Monitoring Performance <s> Recent work suggests that some auto-encoder variants do a good job of capturing the local manifold structure of the unknown data generating density. This paper contributes to the mathematical understanding of this phenomenon and helps define better justified sampling algorithms for deep learning based on auto-encoder variants. We consider an MCMC where each step samples from a Gaussian whose mean and covariance matrix depend on the previous state, defines through its asymptotic distribution a target density. First, we show that good choices (in the sense of consistency) for these mean and covariance functions are the local expected value and local covariance under that target density. Then we show that an auto-encoder with a contractive penalty captures estimators of these local moments in its reconstruction function and its Jacobian. A contribution of this work is thus a novel alternative to maximum-likelihood density estimation, which we call local moment matching. It also justifies a recently proposed sampling algorithm for the Contractive Auto-Encoder and extends it to the Denoising Auto-Encoder. <s> BIB002 </s> Representation Learning: A Review and New Perspectives <s> Evaluating and Monitoring Performance <s> What do auto-encoders learn about the underlying data generating distribution? Recent work suggests that some auto-encoder variants do a good job of capturing the local manifold structure of data. This paper clarifies some of these previous observations by showing that minimizing a particular form of regularized reconstruction error yields a reconstruction function that locally characterizes the shape of the data generating density. We show that the auto-encoder captures the score (derivative of the log-density with respect to the input). It contradicts previous interpretations of reconstruction error as an energy function. Unlike previous results, the theorems provided here are completely generic and do not depend on the parametrization of the auto-encoder: they show what the auto-encoder would tend to if given enough capacity and examples. These results are for a contractive training criterion we show to be similar to the denoising auto-encoder training criterion with small corruption noise, but with contraction applied on the whole reconstruction function rather than just encoder. Similarly to score matching, one can consider the proposed training criterion as a convenient alternative to maximum likelihood because it does not involve a partition function. Finally, we show how an approximate Metropolis-Hastings MCMC can be setup to recover samples from the estimated distribution, and this is confirmed in sampling experiments. <s> BIB003
It is always possible to evaluate a feature learning algorithm in terms of its usefulness with respect to a particular task (e.g., object classification), with a predictor that is fed or initialized with the learned features. In practice, we do this by saving the features learned (e.g., at regular intervals during training, to perform early stopping) and training a cheap classifier on top (such as a linear classifier). However, training the final classifier can be a substantial computational overhead (e.g., supervised fine tuning a deep neural network usually takes more training iterations than the feature learning itself), so we may want to avoid having to train a classifier for every training iteration of the unsupervised learner and every hyperparameter setting. More importantly, this may give an incomplete evaluation of the features (what would happen for other tasks?). All these issues motivate the use of methods to monitor and evaluate purely unsupervised performance. This is rather easy with all the autoencoder variants (with some caution outlined below) and rather difficult with the undirected graphical models such as the RBM and Boltzmann machines. For autoencoder and sparse coding variants, test set reconstruction error can be readily computed, but by itself may be misleading because larger capacity (e.g., more features, more training time) tends to systematically lead to lower reconstruction error, even on the test set. Hence, it cannot be used reliably for selecting most hyperparameters. On the other hand, denoising reconstruction error is clearly immune to this problem, so that solves the problem for DAEs. Based on the connection between DAEs and CAEs uncovered in BIB002 , BIB003 , this immunity can be extended to DAEs, but not to the hyperparameter controlling the amount of noise or of contraction. For RBMs and some (not too deep) Boltzmann machines, one option is the use of annealed importance sampling  to estimate the partition function (and thus the test log likelihood). Note that this estimator can have high variance and that it becomes less reliable (variance becomes too large) as the model becomes more interesting, with larger weights, more nonlinearity, sharper modes, and a sharper probability density function (see our previous discussion in Section 9.4). Another interesting and recently proposed option for RBMs is to track the partition function during training BIB001 , which could be useful for early stopping and reducing the cost of ordinary AIS. For toy RBMs (e.g., 25 hidden units or less, or 25 inputs or less), the exact log likelihood can also be computed analytically, and this can be a good way to debug and verify some properties of interest.