Representation Learning: A Review and New Perspectives <s> Depth and Abstraction <s> A product mixture of a norbornene and a tetracyclododecene is prepared in any desired mole ratio in the range of 95/5 to 5/95 which comprises heating an olefin, cyclopentadiene, and a norbornene in the respective mole ratio of 1-20/1-5/1-5, at a temperature of 100 DEG to 400 DEG C., 100 to 5000 psi, and 0.1 to 5 hours of residence time. <s> BIB001 </s> Representation Learning: A Review and New Perspectives <s> Depth and Abstraction <s> The power of threshold circuits of small depth is investigated. In particular, functions that require exponential-size unweighted threshold circuits of depth 3 when the bottom fan-in is restricted are given. It is proved that there are monotone functions f/sub k/ that can be computed on depth k and linear size AND, OR circuits but require exponential-size to be computed by a depth-(k-1) monotone weighted threshold circuit. > <s> BIB002 </s> Representation Learning: A Review and New Perspectives <s> Depth and Abstraction <s> Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradient based learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank cheque is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal cheques. It is deployed commercially and reads several million cheques per day. <s> BIB003 </s> Representation Learning: A Review and New Perspectives <s> Depth and Abstraction <s> We present a series of theoretical arguments supporting the claim that a large class of modern learning algorithms that rely solely on the smoothness prior - with similarity between examples expressed with a local kernel - are sensitive to the curse of dimensionality, or more precisely to the variability of the target. Our discussion covers supervised, semi-supervised and unsupervised learning algorithms. These algorithms are found to be local in the sense that crucial properties of the learned function at x depend mostly on the neighbors of x in the training set. This makes them sensitive to the curse of dimensionality, well studied for classical non-parametric statistical learning. We show in the case of the Gaussian kernel that when the function to be learned has many variations, these algorithms require a number of training examples proportional to the number of variations, which could be large even though there may exist short descriptions of the target function, i.e. their Kolmogorov complexity may be low. This suggests that there exist non-local learning algorithms that at least have the potential to learn about such structured but apparently complex functions (because locally they have many variations), while not using very specific prior domain knowledge. <s> BIB004 </s> Representation Learning: A Review and New Perspectives <s> Depth and Abstraction <s> Can machine learning deliver AI? Theoretical results, inspiration from the brain and cognition, as well as machine learning experiments suggest that in order to learn the kind of complicated functions that can represent high-level abstractions (e.g. in vision, language, and other AI-level tasks), one would need deep architectures. Deep architectures are composed of multiple levels of non-linear operations, such as in neural nets with many hidden layers, graphical models with many levels of latent variables, or in complicated propositional formulae re-using many sub-formulae. Each level of the architecture represents features at a different level of abstraction, defined as a composition of lower-level features. Searching the parameter space of deep architectures is a difficult task, but new algorithms have been discovered and a new sub-area has emerged in the machine learning community since 2006, following these discoveries. Learning algorithms such as those for Deep Belief Networks and other related unsupervised learning algorithms have recently been proposed to train deep architectures, yielding exciting results and beating the state-of-the-art in certain areas. Learning Deep Architectures for AI discusses the motivations for and principles of learning algorithms for deep architectures. By analyzing and comparing recent results with different learning algorithms for deep architectures, explanations for their success are proposed and discussed, highlighting challenges and suggesting avenues for future explorations in this area. <s> BIB005 </s> Representation Learning: A Review and New Perspectives <s> Depth and Abstraction <s> Deep architectures are families of functions corresponding to deep circuits. Deep Learning algorithms are based on parametrizing such circuits and tuning their parameters so as to approximately optimize some training objective. Whereas it was thought too difficult to train deep architectures, several successful algorithms have been proposed in recent years. We review some of the theoretical motivations for deep architectures, as well as some of their practical successes, and propose directions of investigations to address some of the remaining challenges. <s> BIB006 </s> Representation Learning: A Review and New Perspectives <s> Depth and Abstraction <s> We derive relations between theoretical properties of restricted Boltzmann machines (RBMs), popular machine learning models which form the building blocks of deep learning models, and several natural notions from discrete mathematics and convex geometry. We give implications and equivalences relating RBM-representable probability distributions, perfectly reconstructible inputs, Hamming modes, zonotopes and zonosets, point configurations in hyperplane arrangements, linear threshold codes, and multicovering numbers of hypercubes. As a motivating application, we prove results on the relative representational power of mixtures of product distributions and products of mixtures of pairs of product distributions (RBMs) that formally justify widely held intuitions about distributed representations. In particular, we show that a mixture of products requiring an exponentially larger number of parameters is needed to represent the probability distributions which can be obtained as products of mixtures. <s> BIB007
Depth is a key aspect to the representation learning strategies we consider in this paper. As we will discuss, deep architectures are often challenging to train effectively, and this has been the subject of much recent research and progress. However, despite these challenges, they carry two significant advantages that motivate our long-term interest in discovering successful training strategies for deep architectures. These advantages are: 1) deep architectures promote the reuse of features, and 2) deep architectures can potentially lead to progressively more abstract features at higher layers of representations (more removed from the data). Feature reuse. The notion of reuse, which explains the power of distributed representations, is also at the heart of the theoretical advantages behind deep learning, i.e.,  . Local in the sense that the value of the learned function at x depends mostly on training examples x ðtÞ s close to x. 7. We understand nonparametric as including all learning algorithms whose capacity can be increased appropriately as the amount of data and its complexity demands it, for example, including mixture models and neural networks where the number of parameters is a dataselected hyperparameter. 8. Distributed representations, where k out of N representation elements or feature values can be independently varied; for example, they are not mutually exclusive. Each concept is represented by having k features being turned on or active, while each feature is involved in representing many concepts. 9. Sparse representations: distributed representations where only a few of the elements can be varied at a time, i.e., k < N. 10. As discussed in BIB005 , things are only slightly better when allowing continuous-valued membership values, for example, in ordinary mixture models (with separate parameters for each mixture component), but the difference in representational power is still exponential BIB007 . The situation may also seem better with a decision tree, where each given input is associated with a one-hot code over the tree leaves, which deterministically selects associated ancestors (the path from root to node). Unfortunately, the number of different regions represented (equal to the number of leaves of the tree) still only grows linearly with the number of parameters used to specify it BIB006 . constructing multiple levels of representation or learning a hierarchy of features. The depth of a circuit is the length of the longest path from an input node of the circuit to an output node of the circuit. The crucial property of a deep circuit is that its number of paths, i.e., ways to reuse different parts, can grow exponentially with its depth. Formally, one can change the depth of a given circuit by changing the definition of what each node can compute, but only by a constant factor. The typical computations we allow in each node include: weighted sum, product, artificial neuron model (such as a monotone nonlinearity on top of an affine transformation), computation of a kernel, or logic gates. Theoretical results clearly show families of functions where a deep representation can be exponentially more efficient than one that is insufficiently deep BIB001 , BIB002 , BIB004 ,  , BIB006 . If the same family of functions can be represented with fewer parameters (or more precisely with a smaller VC dimension), learning theory would suggest that it can be learned with fewer examples, yielding improvements in both computational efficiency (less nodes to visit) and statistical efficiency (fewer parameters to learn, and reuse of these parameters over many different kinds of inputs). Abstraction and invariance. Deep architectures can lead to abstract representations because more abstract concepts can often be constructed in terms of less abstract ones. In some cases, such as in the convolutional neural network BIB003 , we explicitly build this abstraction in via a pooling mechanism (see Section 11.2). More abstract concepts are generally invariant to most local changes of the input. That makes the representations that capture these concepts generally highly nonlinear functions of the raw input. This is obviously true of categorical concepts, where more abstract representations detect categories that cover more varied phenomena (e.g., larger manifolds with more wrinkles), and thus they potentially have greater predictive power. Abstraction can also appear in high-level continuous-valued attributes that are only sensitive to some very specific types of changes in the input. Learning these sorts of invariant features has been a long-standing goal in pattern recognition.