Representation Learning: A Review and New Perspectives <s> Distributed Representations <s> The task discussed in this paper is that of learning to map input sequences to output sequences. In particular, problems of phoneme recognition in continuous speech are considered, but most of the discussed techniques could be applied to other tasks, such as the recognition of sequences of handwritten characters. The systems considered in this paper are based on connectionist models, or artificial neural networks, sometimes combined with statistical techniques for recognition of sequences of patterns, stressing the integration of prior knowledge and learning. Different architectures for sequence and speech recognition are reviewed, including recurrent networks as well as hybrid systems involving hidden Markov models. <s> BIB001 </s> Representation Learning: A Review and New Perspectives <s> Distributed Representations <s> A dental flossing aid for dispensing and aiding in the use of dental floss for cleaning teeth, comprises an elongated bodymember on which is mounted a supply of dental floss and which comprises a plurality of holes bored through the bodymember. The bodymember is elongated for projecting into the mouth cavity. Dental floss is threaded through the bored holes of the bodymember and extended to the outermost end of the bodymember; the bores serving to develop tension in the dental floss to resist the removal of floss from the floss supply. A pushkey assembly is provided for overcoming the tension and assisting in advancing the dental floss. <s> BIB002 </s> Representation Learning: A Review and New Perspectives <s> Distributed Representations <s> Can machine learning deliver AI? Theoretical results, inspiration from the brain and cognition, as well as machine learning experiments suggest that in order to learn the kind of complicated functions that can represent high-level abstractions (e.g. in vision, language, and other AI-level tasks), one would need deep architectures. Deep architectures are composed of multiple levels of non-linear operations, such as in neural nets with many hidden layers, graphical models with many levels of latent variables, or in complicated propositional formulae re-using many sub-formulae. Each level of the architecture represents features at a different level of abstraction, defined as a composition of lower-level features. Searching the parameter space of deep architectures is a difficult task, but new algorithms have been discovered and a new sub-area has emerged in the machine learning community since 2006, following these discoveries. Learning algorithms such as those for Deep Belief Networks and other related unsupervised learning algorithms have recently been proposed to train deep architectures, yielding exciting results and beating the state-of-the-art in certain areas. Learning Deep Architectures for AI discusses the motivations for and principles of learning algorithms for deep architectures. By analyzing and comparing recent results with different learning algorithms for deep architectures, explanations for their success are proposed and discussed, highlighting challenges and suggesting avenues for future explorations in this area. <s> BIB003 </s> Representation Learning: A Review and New Perspectives <s> Distributed Representations <s> While vector quantization (VQ) has been applied widely to generate features for visual recognition problems, much recent work has focused on more powerful methods. In particular, sparse coding has emerged as a strong alternative to traditional VQ approaches and has been shown to achieve consistently higher performance on benchmark datasets. Both approaches can be split into a training phase, where the system learns a dictionary of basis functions, and an encoding phase, where the dictionary is used to extract features from new inputs. In this work, we investigate the reasons for the success of sparse coding over VQ by decoupling these phases, allowing us to separate out the contributions of training and encoding in a controlled way. Through extensive experiments on CIFAR, NORB and Caltech 101 datasets, we compare several training and encoding schemes, including sparse coding and a form of VQ with a soft threshold activation function. Our results show not only that we can use fast VQ algorithms for training, but that we can just as well use randomly chosen exemplars from the training set. Rather than spend resources on training, we find it is more important to choose a good encoder—which can often be a simple feed forward non-linearity. Our results include state-of-the-art performance on both CIFAR and NORB. <s> BIB004
Good representations are expressive, meaning that a reasonably sized learned representation can capture a huge number of possible input configurations. A simple counting argument helps us to assess the expressiveness of a model producing a representation: How many parameters does it require compared to the number of input regions (or configurations) it can distinguish? Learners of one-hot representations, such as traditional clustering algorithms, Gaussian mixtures, nearest-neighbor algorithms, decision trees, or Gaussian SVMs, all require OðNÞ parameters (and/ or OðNÞ examples) to distinguish OðNÞ input regions. One could naively believe that one cannot do better. However, restricted Boltzmann machines (RBMs), sparse coding, autoencoders, or multilayer neural networks can all represent up to Oð2 k Þ input regions using only OðNÞ parameters (with k the number of nonzero elements in a sparse representation, and k ¼ N in nonsparse RBMs and other dense representations). These are all distributed 8 or sparse BIB001 representations. The generalization of clustering to distributed representations is multiclustering, where either several clusterings take place in parallel or the same clustering is applied on different parts of the input, such as in the very popular hierarchical feature extraction for object recognition based on a histogram of cluster categories detected in different patches of an image  , BIB004 . The exponential gain from distributed or sparse representations is discussed further in BIB003 Section 3.2, Fig. 3.2] . It comes about because each parameter (e.g., the parameters of one of the units in a sparse code or one of the units in an RBM) can be reused in many examples that are not simply near neighbors of each other, whereas with local generalization, different regions in input space are basically associated with their own private set of parameters, for example, as in decision trees, nearest neighbors, Gaussian SVMs, and so on. In a distributed representation, an exponentially large number of possible subsets of features or hidden units can be activated in response to a given input. In a single-layer model, each feature is typically associated with a preferred input direction, corresponding to a hyperplane in input space, and the code or representation associated with that input is precisely the pattern of activation (which features respond to the input, and how much). This is in contrast with a nondistributed representation such as the one learned by most clustering algorithms, for example, k-means, in which the representation of a given input vector is a one-hot code identifying which one of a small number of cluster centroids best represents the input. BIB002 