Representation Learning: A Review and New Perspectives <s> CONCLUSION <s> Several recent advances to the state of the art in image classification benchmarks have come from better configurations of existing techniques rather than novel approaches to feature learning. Traditionally, hyper-parameter optimization has been the job of humans because they can be very efficient in regimes where only a few trials are possible. Presently, computer clusters and GPU processors make it possible to run more trials and we show that algorithmic approaches can find better results. We present hyper-parameter optimization results on tasks of training neural networks and deep belief networks (DBNs). We optimize hyper-parameters using random search and two new greedy sequential methods based on the expected improvement criterion. Random search has been shown to be sufficiently efficient for learning neural networks for several datasets, but we show it is unreliable for training DBNs. The sequential algorithms are applied to the most difficult DBN learning problems from [1] and find significantly better results than the best previously reported. This work contributes novel techniques for making response surface models P(y|x) in which many elements of hyper-parameter assignment (x) are known to be irrelevant given particular values of other elements. <s> BIB001 </s> Representation Learning: A Review and New Perspectives <s> CONCLUSION <s> Learning algorithms related to artificial neural networks and in particular for Deep Learning may seem to involve many bells and whistles, called hyper-parameters. This chapter is meant as a practical guide with recommendations for some of the most commonly used hyperparameters, in particular in the context of learning algorithms based on back-propagated gradient and gradient-based optimization. It also discusses how to deal with the fact that more interesting results can be obtained when allowing one to adjust many hyper-parameters. Overall, it describes elements of the practice used to successfully and efficiently train and debug large-scale and often deep multi-layer neural networks. It closes with open questions about the training difficulties observed with deeper architectures. <s> BIB002 </s> Representation Learning: A Review and New Perspectives <s> CONCLUSION <s> Grid search and manual search are the most widely used strategies for hyper-parameter optimization. This paper shows empirically and theoretically that randomly chosen trials are more efficient for hyper-parameter optimization than trials on a grid. Empirical evidence comes from a comparison with a large previous study that used grid search and manual search to configure neural networks and deep belief networks. Compared with neural networks configured by a pure grid search, we find that random search over the same domain is able to find models that are as good or better within a small fraction of the computation time. Granting random search the same computational budget, random search finds better models by effectively searching a larger, less promising configuration space. Compared with deep belief networks configured by a thoughtful combination of manual search and grid search, purely random search over the same 32-dimensional configuration space found statistically equal performance on four of seven data sets, and superior performance on one of seven. A Gaussian process analysis of the function from hyper-parameters to validation set performance reveals that for most data sets only a few of the hyper-parameters really matter, but that different hyper-parameters are important on different data sets. This phenomenon makes grid search a poor choice for configuring algorithms for new data sets. Our analysis casts some light on why recent "High Throughput" methods achieve surprising success--they appear to search through a large number of hyper-parameters because most hyper-parameters do not matter much. We anticipate that growing interest in large hierarchical models will place an increasing burden on techniques for hyper-parameter optimization; this work shows that random search is a natural baseline against which to judge progress in the development of adaptive (sequential) hyper-parameter optimization algorithms. <s> BIB003 </s> Representation Learning: A Review and New Perspectives <s> CONCLUSION <s> The use of machine learning algorithms frequently involves careful tuning of learning parameters and model hyperparameters. Unfortunately, this tuning is often a "black art" requiring expert experience, rules of thumb, or sometimes brute-force search. There is therefore great appeal for automatic approaches that can optimize the performance of any given learning algorithm to the problem at hand. In this work, we consider this problem through the framework of Bayesian optimization, in which a learning algorithm's generalization performance is modeled as a sample from a Gaussian process (GP). We show that certain choices for the nature of the GP, such as the type of kernel and the treatment of its hyperparameters, can play a crucial role in obtaining a good optimizer that can achieve expertlevel performance. We describe new algorithms that take into account the variable cost (duration) of learning algorithm experiments and that can leverage the presence of multiple cores for parallel experimentation. We show that these proposed algorithms improve on previous automatic procedures and can reach or surpass human expert-level optimization for many algorithms including latent Dirichlet allocation, structured SVMs and convolutional neural networks. <s> BIB004 </s> Representation Learning: A Review and New Perspectives <s> CONCLUSION <s> Recent work suggests that some auto-encoder variants do a good job of capturing the local manifold structure of the unknown data generating density. This paper contributes to the mathematical understanding of this phenomenon and helps define better justified sampling algorithms for deep learning based on auto-encoder variants. We consider an MCMC where each step samples from a Gaussian whose mean and covariance matrix depend on the previous state, defines through its asymptotic distribution a target density. First, we show that good choices (in the sense of consistency) for these mean and covariance functions are the local expected value and local covariance under that target density. Then we show that an auto-encoder with a contractive penalty captures estimators of these local moments in its reconstruction function and its Jacobian. A contribution of this work is thus a novel alternative to maximum-likelihood density estimation, which we call local moment matching. It also justifies a recently proposed sampling algorithm for the Contractive Auto-Encoder and extends it to the Denoising Auto-Encoder. <s> BIB005 </s> Representation Learning: A Review and New Perspectives <s> CONCLUSION <s> It has been hypothesized, and supported with experimental evidence, that deeper representations, when well trained, tend to do a better job at disentangling the underlying factors of variation. We study the following related conjecture: better representations, in the sense of better disentangling, can be exploited to produce Markov chains that mix faster between modes. Consequently, mixing between modes would be more efficient at higher levels of representation. To better understand this, we propose a secondary conjecture: the higher-level samples fill more uniformly the space they occupy and the high-density manifolds tend to unfold when represented at higher levels. The paper discusses these hypotheses and tests them experimentally through visualization and measurements of mixing between modes and interpolating between samples. <s> BIB006
This review of representation learning and deep learning has covered three major and apparently disconnected approaches: the probabilistic models (both the directed kind such as sparse coding and the undirected kind such as Boltzmann machines), the reconstruction-based algorithms related to autoencoders, and the geometrically motivated manifold-learning approaches. Drawing connections between these approaches is currently a very active area of research and is likely to continue to produce models and methods that take advantage of the relative strengths of each paradigm. Practical concerns and guidelines. One of the criticisms addressed to artificial neural networks and deep learning algorithms is that they have many hyperparameters and variants and that exploring their configurations and architectures is an art. This has motivated an earlier book on the "Tricks of the Trade"  of which  is still relevant for training deep architectures, in particular what concerns initialization, ill conditioning, and stochastic gradient descent. A good and more modern compendium of good training practice, particularly adapted to training RBMs, is provided in  , while a similar guide oriented more toward deep neural networks can be found in BIB002 , both of which are part of a novel version of the above book. Recent work on automating hyperparameter search BIB003 , BIB001 , BIB004 is also making it more convenient, efficient, and reproducible. Incorporating generic AI-level priors. We have covered many high-level generic priors that we believe could bring machine learning closer to AI by improving representation learning. Many of these priors relate to the assumed existence of multiple underlying factors of variation, whose variations are in some sense orthogonal to each other. They are expected to be organized at multiple levels of abstraction; hence the need for deep architectures, which also have statistical advantages because they allow to reuse parameters in a combinatorially efficient way. Only a few of these factors would typically be relevant for any particular example, justifying sparsity of representation. These factors are expected to be related to simple (e.g., linear) dependencies, with subsets of these explaining different random variables of interest (inputs, tasks) and varying in structured ways in time and space (temporal and spatial coherence). We expect future successful applications of representation learning to refine and increase that list of priors, and to incorporate most of them instead of focusing on only one. Research in training criteria that better take these priors into account is likely to move us closer to the long-term objective of discovering learning algorithms that can disentangle the underlying explanatory factors. Inference. We anticipate that methods based on directly parameterizing a representation function will incorporate more and more of the iterative type of computation one finds in the inference procedures of probabilistic latentvariable models. There is already movement in the other direction, with probabilistic latent-variable models exploiting approximate inference mechanisms that are themselves learned (i.e., producing a parametric description of the representation function). A major appeal of probabilistic models is that the semantics of the latent variables are clear, and this allows a clean separation of the problems of modeling (choose the energy function), inference (estimating P ðh j xÞ), and learning (optimizing the parameters), using generic tools in each case. On the other hand, doing approximate inference and not taking BIB005 . The changes that affect class identity might, in input space, actually be of similar magnitude to local deformations, but not follow along the manifold, i.e., cross zones of low density. BIB006 . As an aside, if we are given only the values of the higher level pooling features, we cannot accurately recover the data because we do not know how to apportion credit for the pooling feature values to the lower level features. This is simply the generative version of the consequences of the loss of information caused by pooling. that approximation into account explicitly in the approximate optimization for learning could have detrimental effects; hence the appeal of learning approximate inference. More fundamentally, there is the question of the multimodality of the posterior P ðh j xÞ. If there are exponentially many probable configurations of values of the factors h i that can explain x, then we seem to be stuck with very poor inference, either focusing on a single mode (MAP inference), assuming some kind of strong factorization (as in variational inference), or using an MCMC that cannot visit enough modes of P ðh j xÞ. What we propose as food for thought is the idea of dropping the requirement of an explicit representation of the posterior and settling for an implicit representation that exploits potential structure in P ðh j xÞ to represent it compactly: Even though P ðh j xÞ may have an exponential number of modes, it may be possible to represent it with a small set of numbers. For example, consider computing a deterministic feature representation fðxÞ that implicitly captures the information about a highly multimodal P ðh j xÞ, in the sense that all the questions (e.g., making some prediction about some target concept) that can be asked from P ðh j xÞ can also be answered from fðxÞ. Optimization. Much remains to be done to better understand the successes and failures of training deep architectures, both in the supervised case (with many recent successes) and the unsupervised case (where much more work needs to be done). Although regularization effects can be important on small datasets, the effects that persist on very large datasets suggest some optimization issues are involved. Are they more due to local minima (we now know there are huge numbers of them) and the dynamics of the training procedure? Or are they due mostly to ill conditioning and may be handled by approximate secondorder methods? These basic questions remain unanswered and deserve much more study. Pascal Vincent is an associate professor in the Department of Computer Science and Operations Research at the University of Montreal, and a CIFAR associate in the Neural Computation and Adaptive Perception program. His recent research has focused on novel principles for learning direct encodings of representations. He is also interested in developing alternative parameter estimation techniques for probabilistic models. . For more information on this or any other computing topic, please visit our Digital Library at www.computer.org/publications/dlib.