Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> INTRODUCTION <s> A simple linear neuron model with constrained Hebbian-type synaptic modification is analyzed and a new class of unconstrained learning rules is derived. It is shown that the model neuron tends to extract the principal component from a stationary input vector sequence. <s> BIB001 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> INTRODUCTION <s> This paper reports the results of our studies with an unsupervised learning paradigm which we have called “Competitive Learning.” We have examined competitive learning using both computer simulation and formal analysis and have found that when it is applied to parallel networks of neuron-like elements, many potentially useful learning tasks can be accomplished. We were attracted to competitive learning because it seems to provide a way to discover the salient, general features which can be used to classify a set of patterns. We show how a very simply competitive mechanism can discover a set of feature detectors which capture important aspects of the set of stimulus input patterns. We also show how these feature detectors can form the basis of a multilayer system that can serve to learn categorizations of stimulus sets which are not linearly separable. We show how the use of correlated stimuli con serve as a kind of “teaching” input to the system to allow the development of feature detectors which would not develop otherwise. Although we find the competitive learning mechanism a very interesting and powerful learning principle, we do not, of course, imagine thot it is the only learning principle. Competitive learning is an essentially nonassociative statistical learning scheme. We certainly imagine that other kinds of learning mechanisms will be involved in the building of associations among patterns of activation in a more complete neural network. We offer this analysis of these competitive learning mechanisms to further our understanding of how simple adaptive networks can discover features important in the description of the stimulus environment in which the system finds itself. <s> BIB002 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> INTRODUCTION <s> Abstract : A parallel distributed processing model of visual word recognition and pronunciation and of the acquisition of these skills is described. The model consists of a set of orthographic units used to code letter strings, a set of hidden units, and a set of phonemic units. Weights on connections between units were modified during a training phase using the back-propagation learning algorithm. The model takes letter strings as input and yields two types of output: a pattern of activation across the phonemic units, and a recreation of the input spelling pattern across the orthographic units. The model was trained on a corpus of 2897 English words that included most of the uninflected monosyllabic words in the language. The model provides detailed accounts of performance on two tasks, naming aloud and lexical decision, and simulates many aspects of human performance, including (a) differences between words in terms of processing difficulty; (b) pronunciation of novel items; (c) differences between readers in terms of word recognition skill; (d) transitions from beginning to skilled reading; and (e) differences in performance on the two tasks. The model's behavior early in the learning phase corresponds to that of children acquiring word recognition skills. Training with a smaller number of hidden units produces output characteristic of many poor readers. (kr) <s> BIB003 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> INTRODUCTION <s> Abstract Deep dyslexia is an acquired reading disorder marked by the Occurrence of semantic errors (e.g. reading RIVER as “ocean”). In addition, patients exhibit a number of other symptoms, including visual and morphological effects in their errors, a part-of-speech effect, and an advantage for concrete over abstract words. Deep dyslexia poses a distinct challenge for cognitive neuropsychology because there is little understanding of why such a variety of symptoms should co-occur in virtually all known patients. Hinton and Shallice (1991) replicated the co-occurrence of visual and semantic errors by lesioning a recurrent connectionist network trained to map from orthography to semantics. Although the success of their simulations is encouraging. there is little understanding of what underlying principles are responsible for them. In this paper we evaluate and, where possible, improve on the most important design decisions made by Hinton and Shallice, relating to the task, the network architecture, the trai... <s> BIB004 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> INTRODUCTION <s> Discovering the structure inherent in a set of patterns is a fundamental aim of statistical inference or learning. One fruitful approach is to build a parameterized stochastic generative model, independent draws from which are likely to produce the patterns. For all but the simplest generative models, each pattern can be generated in exponentially many ways. It is thus intractable to adjust the parameters to maximize the probability of the observed patterns. We describe a way of finessing this combinatorial explosion by maximizing an easily computed lower bound on the probability of the observations. Our method can be viewed as a form of hierarchical self-supervised learning that may relate to the function of bottom-up and top-down cortical processing pathways. <s> BIB005 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> INTRODUCTION <s> We develop a connectionist approach to processing in quasi-regular domains, as exemplifiedby English word reading. A consideration of the shortcomings of a previous implementation (Seidenberg & McClelland, 1989, Psych. Rev.) in reading nonwords leads to the development of orthographic and phonological representations that capture better the relevant structure among the written and spoken forms of words. In a number of simulation experiments, networks using the new representations learn to read both regular and exception words, including low-frequency exception words, and yet are still able to read pronounceable nonwords as well as skilled readers. A mathematical analysis of the effects of word frequency and spelling-sound consistency in a related but simpler system serves to clarify the close relationship of these factors in influencing naming latencies. These insights are verifiedin subsequent simulations, including an attractor network that reproduces the naming latency data directly in its time to settle on a response. Further analyses of the network’ s ability to reproduce data on impaired reading in surface dyslexia support a view of the reading system that incorporates a graded division-of-labor between semantic and phonological processes. Such a view is consistent with the more general Seidenberg and McClelland framework and has some similarities with—b ut also important differences from—the standard dual-route account. <s> BIB006 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> INTRODUCTION <s> We describe a hierarchical, generative model that can be viewed as a nonlinear generalization of factor analysis and can be implemented in a neural network. The model uses bottom–up, top–down and lateral connections to perform Bayesian perceptual inference correctly. Once perceptual inference has been performed the connection strengths can be updated using a very simple learning rule that only requires locally available information. We demonstrate that the network learns to extract sparse, distributed, hierarchical representations. <s> BIB007 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> INTRODUCTION <s> A connectionist study of word reading is described that emphasizes the computational demands of the spelling-sound mapping in determining the properties of the reading system. It is shown that the phonological assembly process can be implemented by a two-layer network, which easily extracts the regularities in the spelling-sound mapping for English from training data containing many exception words. It is argued that productive knowledge about spelling-sound relationships is more easily acquired and used if it is separated from case-specific knowledge of the pronunciation of known words. It is then shown how the interaction of assembled and retrieved phonologies can account for the combined effects of frequency and regularity-consistency and for the reading performance of dyslexic patients. It is concluded that the organization of the reading system reflects the demands of the task and that the pronunciations of nonwords and exception words are computed by different processes. <s> BIB008 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> INTRODUCTION <s> Abstract This review describes and motivates six principles for computational cognitive neuroscience models: biological realism, distributed representations, inhibitory competition, bidirectional activation propagation, error-driven task learning, and Hebbian model learning. Although these principles are supported by a number of cognitive, computational and biological motivations, the prototypical neural-network model (a feedforward back-propagation network) incorporates only two of them, and no widely used model incorporates all of them. It is argued here that these principles should be integrated into a coherent overall framework, and some potential synergies and conflicts in doing so are discussed. <s> BIB009 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> INTRODUCTION <s> Since its founding in 1989 by Terrence Sejnowski, Neural Computation has become the leading journal in the field. Foundations of Neural Computationcollects, by topic, the most significant papers that have appeared in the journal over the past nine years.This volume of Foundations of Neural Computation, on unsupervised learning algorithms, focuses on neural network learning algorithms that do not require an explicit teacher. The goal of unsupervised learning is to extract an efficient internal representation of the statistical structure implicit in the inputs. These algorithms provide insights into the development of the cerebral cortex and implicit learning in humans. They are also of interest to engineers working in areas such as computer vision and speech recognition who seek efficient representations of raw input data. <s> BIB010 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> INTRODUCTION <s> A persistent worry with computational models of unsupervised learning is that learning will become more difficult as the problem is scaled. We examine this issue in the context of a novel hierarchical, generative model that can be viewed as a nonlinear generalisation of factor analysis and can be implemented in a neural network. The model performs perceptual inference in a probabilistically consistent manner by using top-down, bottom-up and lateral connections. These connections can be learned using simple rules that require only locally available information. We first demonstrate that the model can extract a sparse, distributed, hierarchical representation of global disparity from simplified random-dot stereograms. We then investigate some of the scaling properties of the algorithm on this problem and find that: 1) increasing the image size leads to faster and more reliable learning; 2) increasing the depth of the network from one to two hidden layers leads to better representations at the first hidden layer; and 3) once one part of the network has discovered how to represent disparity, it "supervises" other parts of the network, greatly speeding up their learning. <s> BIB011 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> INTRODUCTION <s> We describe a model of visual processing in which feedback connections from a higher- to a lower- order visual cortical area carry predictions of lower-level neural activities, whereas the feedforward connections carry the residual errors between the predictions and the actual lower-level activities. When exposed to natural images, a hierarchical network of model neurons implementing such a model developed simple-cell-like receptive fields. A subset of neurons responsible for carrying the residual errors showed endstopping and other extra-classical receptive-field effects. These results suggest that rather than being exclusively feedforward phenomena, nonclassical surround effects in the visual cortex may also result from cortico-cortical feedback as a consequence of the visual system using an efficient hierarchical strategy for encoding natural images. <s> BIB012 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> INTRODUCTION <s> Two unsupervised learning modes (incidental and intentional unsupervised learning) and their relation to supervised classification learning are examined. The approach allows for direct comparisons of unsupervised learning data with the Shepard, Hovland, and Jenkins (1961) seminal studies in supervised classification learning. Unlike supervised classification learning, unsupervised learning (especially under incidental conditions) favors linear category structures over compact nonlinear category structures. Unsupervised learning is shown to be multifaceted in that performance varies with task conditions. In comparison with incidental unsupervised learning, intentional unsupervised learning is more rule like, but is no more accurate. The acquisition and application of knowledge is also more laborious under intentional unsupervised learning. <s> BIB013 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> INTRODUCTION <s> SUSTAIN (Supervised and Unsupervised STratified Adaptive Incremental Network) is a model of how humans learn categories from examples. SUSTAIN initially assumes a simple category structure. If simple solutions prove inadequate and SUSTAIN is confronted with a surprising event (e.g., it is told that a bat is a mammal instead of a bird), SUSTAIN recruits an additional cluster to represent the surprising event. Newly recruited clusters are available to explain future events and can themselves evolve into prototypes-attractors-rules. SUSTAIN's discovery of category substructure is affected not only by the structure of the world but by the nature of the learning task and the learner's goals. SUSTAIN successfully extends category learning models to studies of inference learning, unsupervised learning, category construction, and contexts in which identification learning is faster than classification learning. <s> BIB014 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> INTRODUCTION <s> This article concerns the nature of evoked brain responses and the principles underlying their generation. We start with the premise that the sensory brain has evolved to represent or infer the causes of changes in its sensory inputs. The problem of inference is well formulated in statistical terms. The statistical fundaments of inference may therefore afford important constraints on neuronal implementation. By formulating the original ideas of Helmholtz on perception, in terms of modern-day statistical theories, one arrives at a model of perceptual inference and learning that can explain a remarkable range of neurobiological facts. It turns out that the problems of inferring the causes of sensory input (perceptual inference) and learning the relationship between input and cause (perceptual learning) can be resolved using exactly the same principle. Specifically, both inference and learning rest on minimizing the brain’s free energy, as defined in statistical physics. Furthermore, inference and learning can proceed in a biologically plausible fashion. Cortical responses can be seen as the brain’s attempt to minimize the free energy induced by a stimulus and thereby encode the most likely cause of that stimulus. Similarly, learning emerges from changes in synaptic efficacy that minimize the free energy, averaged over all stimuli encountered. The underlying scheme rests on empirical Bayes and hierarchical models of how sensory input is caused. The use of hierarchical models enables the brain to construct prior expectations in a dynamic and context-sensitive fashion. This scheme provides a principled way to understand many aspects of cortical organization and responses. The aim of this article is to encompass many apparently unrelated anatomical, physiological and psychophysical attributes of the brain within a single theoretical perspective. In terms of cortical architectures, the theoretical treatment predicts that sensory cortex should be arranged hierarchically, that connections should be reciprocal and that forward and backward connections should show a functional asymmetry (forward connections are driving, whereas backward connections are both driving and modulatory). In terms of synaptic physiology, it predicts associative plasticity and, for dynamic models, spike-timing-dependent plasticity. In terms of electrophysiology, it accounts for classical and extra classical receptive field effects and long-latency or endogenous components of evoked cortical responses. It predicts the attenuation of responses encoding prediction error with perceptual learning and explains many phenomena such as repetition suppression, mismatch negativity (MMN) and the P300 in electroencephalography. In psychophysical terms, it accounts for the behavioural correlates of these physiological phenomena, for example, priming and global precedence. The final focus of this article is on perceptual learning as measured with the MMN and the implications for empirical studies of coupling among cortical areas using evoked sensory responses. <s> BIB015 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> INTRODUCTION <s> How is reading, a cultural invention, coded by neural populations in the human brain? The neural code for written words must be abstract, because we can recognize words regardless of their location, font and size. Yet it must also be exquisitely sensitive to letter identity and letter order. Most existing coding schemes are insufficiently invariant or incompatible with the constraints of the visual system. We propose a tentative neuronal model according to which part of the occipito-temporal 'what' pathway is tuned to writing and forms a hierarchy of local combination detectors sensitive to increasingly larger fragments of words. Our proposal can explain why the detection of 'open bigrams' (ordered pairs of letters) constitutes an important stage in visual word recognition. <s> BIB016 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> INTRODUCTION <s> High-dimensional data can be converted to low-dimensional codes by training a multilayer neural network with a small central layer to reconstruct high-dimensional input vectors. Gradient descent can be used for fine-tuning the weights in such “autoencoder” networks, but this works well only if the initial weights are close to a good solution. We describe an effective way of initializing the weights that allows deep autoencoder networks to learn low-dimensional codes that work much better than principal components analysis as a tool to reduce the dimensionality of data. <s> BIB017 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> INTRODUCTION <s> At least 3 different types of computational model have been shown to account for various facets of both normal and impaired single word reading: (a) the connectionist triangle model, (b) the dual-route cascaded model, and (c) the connectionist dual process model. Major strengths and weaknesses of these models are identified. In the spirit of nested incremental modeling, a new connectionist dual process model (the CDP model) is presented. This model builds on the strengths of 2 of the previous models while eliminating their weaknesses. Contrary to the dual-route cascaded model, CDP is able to learn and produce graded consistency effects. Contrary to the triangle and the connectionist dual process models, CDP accounts for serial effects and has more accurate nonword reading performance. CDP also beats all previous models by an order of magnitude when predicting individual item-level variance on large databases. Thus, the authors show that building on existing theories by combining the best features of previous models—a nested modeling strategy that is commonly used in other areas of science but often neglected in psychology—results in better and more powerful computational models. <s> BIB018 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> INTRODUCTION <s> To achieve its impressive performance in tasks such as speech perception or object recognition, the brain extracts multiple levels of representation from the sensory input. Backpropagation was the first computationally efficient model of how neural networks could learn multiple layers of representation, but it required labeled training data and it did not work well in deep networks. The limitations of backpropagation learning can now be overcome by using multilayer neural networks that contain top-down connections and training them to generate sensory data rather than to classify it. Learning multilayer generative models might seem difficult, but a recent discovery makes it easy to learn nonlinear distributed representations one layer at a time. <s> BIB019 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> INTRODUCTION <s> Visual word recognition has been proposed to rely on a hierarchy of increasingly complex neuronal detectors, from individual letters to bigrams and morphemes. We used fMRI to test whether such a hierarchy is present in the left occipitotemporal cortex, at the site of the visual word-form area, and with an anterior-to-posterior progression. We exposed adult readers to (1) false-font strings; (2) strings of infrequent letters; (3) strings of frequent letters but rare bigrams; (4) strings with frequent bigrams but rare quadrigrams; (5) strings with frequent quadrigrams; (6) real words. A gradient of selectivity was observed through the entire span of the occipitotemporal cortex, with activation becoming more selective for higher-level stimuli toward the anterior fusiform region. A similar gradient was also seen in left inferior frontoinsular cortex. Those gradients were asymmetrical in favor of the left hemisphere. We conclude that the left occipitotemporal visual word-form area, far from being a homogeneous structure, presents a high degree of functional and spatial hierarchical organization which must result from a tuning process during reading acquisition. <s> BIB020 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> INTRODUCTION <s> Most words in English have more than one syllable, yet the most influential computational models of reading aloud are restricted to processing monosyllabic words. Here, we present CDP++, a new version of the Connectionist Dual Process model (Perry, Ziegler, & Zorzi, 2007). CDP++ is able to simulate the reading aloud of mono- and disyllabic words and nonwords, and learns to assign stress in exactly the same way as it learns to associate graphemes with phonemes. CDP++ is able to simulate the monosyllabic benchmark effects its predecessor could, and therefore shows full backwards compatibility. CDP++ also accounts for a number of novel effects specific to disyllabic words, including the effects of stress regularity and syllable number. In terms of database performance, CDP++ accounts for over 49% of the reaction time variance on items selected from the English Lexicon Project, a very large database of several thousand of words. With its lexicon of over 32,000 words, CDP++ is therefore a notable example of the successful scaling-up of a connectionist model to a size that more realistically approximates the human lexical system. <s> BIB021 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> INTRODUCTION <s> One of the central problems in computational neuroscience is to understand how the object-recognition pathway of the cortex learns a deep hierarchy of nonlinear feature detectors. Recent progress in machine learning shows that it is possible to learn deep hierarchies without requiring any labelled data. The feature detectors are learned one layer at a time and the goal of the learning procedure is to form a good generative model of images, not to predict the class of each image. The learning procedure only requires the pairwise correlations between the activations of neuron-like processing units in adjacent layers. The original version of the learning procedure is derived from a quadratic 'energy' function but it can be extended to allow third-order, multiplicative interactions in which neurons gate the pairwise interactions between other neurons. A technique for factoring the third-order interactions leads to a learning module that again has a simple learning rule based on pairwise correlations. This module looks remarkably like modules that have been proposed by both biologists trying to explain the responses of neurons and engineers trying to create systems that can recognize objects. <s> BIB022 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> INTRODUCTION <s> Numerosity estimation is phylogenetically ancient and foundational to human mathematical learning, but its computational bases remain controversial. Here we show that visual numerosity emerges as a statistical property of images in 'deep networks' that learn a hierarchical generative model of the sensory input. Emergent numerosity detectors had response profiles resembling those of monkey parietal neurons and supported numerosity estimation with the same behavioral signature shown by humans and animals. <s> BIB023 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> INTRODUCTION <s> It is often assumed that graphemes are a crucial level of orthographic representation above letters. Current connectionist models of reading, however, do not address how the mapping from letters to graphemes is learned. One major challenge for computational modeling is therefore developing a model that learns this mapping and can assign the graphemes to linguistically meaningful categories such as the onset, vowel, and coda of a syllable. Here, we present a model that learns to do this in English for strings of any letter length and any number of syllables. The model is evaluated on error rates and further validated on the results of a behavioral experiment designed to examine ambiguities in the processing of graphemes. The results show that the model (a) chooses graphemes from letter strings with a high level of accuracy, even when trained on only a small portion of the English lexicon; (b) chooses a similar set of graphemes as people do in situations where different graphemes can potentially be selected; (c) predicts orthographic effects on segmentation which are found in human data; and (d) can be readily integrated into a full-blown model of multi-syllabic reading aloud such as CDP++ (Perry, Ziegler, & Zorzi, 2010). Altogether, these results suggest that the model provides a plausible hypothesis for the kind of computations that underlie the use of graphemes in skilled reading. <s> BIB024 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> INTRODUCTION <s> It is possible to learn multiple layers of non-linear features by backpropagating error derivatives through a feedforward neural network. This is a very effective learning procedure when there is a huge amount of labeled training data, but for many learning tasks very few labeled examples are available. In an effort to overcome the need for labeled data, several different generative models were developed that learned interesting features by modeling the higher order statistical structure of a set of input vectors. One of these generative models, the restricted Boltzmann machine (RBM), has no connections between its hidden units and this makes perceptual inference and learning much simpler. More significantly, after a layer of hidden features has been learned, the activities of these features can be used as training data for another RBM. By applying this idea recursively, it is possible to learn a deep hierarchy of progressively more complicated features without requiring any labeled data. This deep hierarchy can then be treated as a feedforward neural network which can be discriminatively fine-tuned using backpropagation. Using a stack of RBMs to initialize the weights of a feedforward neural network allows backpropagation to work effectively in much deeper networks and it leads to much better generalization. A stack of RBMs can also be used to initialize a deep Boltzmann machine that has many hidden layers. Combining this initialization method with a new method for fine-tuning the weights finally leads to the first efficient way of training Boltzmann machines with many hidden layers and millions of weights. <s> BIB025
A fundamental issue in the study of human cognition is what computations are carried out by the brain to implement cognitive processes. The connectionist framework assumes that cognitive processes are implemented in terms of complex, non-linear interactions among a large number of simple, neuron-like processing units that form a neural network . This approach has been used in cognitive psychologyoften with success-to develop functional models that clearly represent a great advance over previous verbal-diagrammatic models because they can produce simulations of learning, skilled performance, and breakdowns of processing after brain damage. One paradigmatic example is the connectionist modeling of visual word recognition and reading aloud, which has often provided key theoretical and methodological advances with broad influences well-beyond the language domain (e.g.,  BIB003 BIB004 BIB006 . Connectionist models of the reading processes can produce highly detailed simulations of human performance, accounting for a wide range of empirical data that include reaction times and accuracy of skilled readers at the level of individual words, the development of reading skills in children, and the impaired performance of dyslexic individuals BIB006 BIB008 Seidenberg, 1999, 2004; BIB018 BIB021 BIB024 . Despite significant progress in the attempt to improve the architectural and learning principles incorporated in neural network models (see BIB009  , much modeling work in psychology is still based on the classic neural network with one layer of hidden units (i.e., a "shallow" architecture) and error backpropagation as learning algorithm-a choice that is typically seen as a compromise to achieve efficient learning of complex cognitive tasks. We argue below that a key step forward for connectionist modeling is the use of networks with a "deep" architecture BIB019 BIB025 and where most of the learning is generative rather than discriminative (Box 1). The shallow architecture of the prototypical multi-layer neural network does not capture the hierarchical organization of the cerebral cortex. Hierarchical processing is thought to be a fundamental characteristic of cortical computation BIB019  and it is a key feature of biologically inspired computational models of vision  . The idea of a deep network with a hierarchy of increasingly complex feature detectors can be traced back to the Interactive Activation Model (IAM) of letter and word perception  , but this seminal proposal did not transfer to connectionist learning models because the error backpropagation algorithm had little success in training networks with many hidden layers BIB019 BIB025 . Another key assumption of the IAM that did not readily transfer to connectionist learning models is the mixing of bottom-up and top-down processing through recurrent feedback. Finally, the widespread use of the error backpropagation algorithm in connectionist modeling, leaving aside its lack of biological plausibility BIB009 , implies subscription to the dubious assumption that learning is largely discriminative (e.g., classification or function learning) and that an external teaching signal is available at each learning event (that is, all training data is labeled). This learning regimen is exceptional in the real world. Reinforcement learning  ) is a plausible alternative, but there is a broad range of situations where learning is fully unsupervised and its only objective is that of building rich internal representations of the sensory world BIB010 . Notably, the learned internal model can then be used to infer causes and make predictions BIB005 BIB007 BIB015 BIB022   . Unsupervised learning has a long history, but the classic learning algorithms have important limitations. Some develop a representation that is distributed but also linear BIB001 , which implies that higher-order information remains invisible. Others develop a representation that is non-linear but also localist, that is one in which each observation is associated to a single hidden unit BIB002  . For these reasons, their application to modeling complex cognitive functions has been limited. An important breakthrough in unsupervised learning is the use of statistical principles such as maximum likelihood and Bayesian estimation to develop generative models that discover representations that are both distributed and non-linearly related to the input data BIB007 . A generative model is a probabilistic model that captures the hidden (latent) causes of the data, thereby providing a sensible objective function for unsupervised learning. In other words, the "learner" estimates a model, without any supervision or reward, that represents the probability distribution of the data. Generative models are appealing because they make strong suggestions about the role of feedback connections in the cortex and are consistent with neurobiological theories that emphasize the mixing of bottom-up and top-down interactions in the brain: bottom-up inputs convey sensory information, whereas internal representations form a generative model that predicts the sensory input via top-down activation BIB007 . Learning can be viewed as maximizing the likelihood of the observed data under the generative model, which is equivalent to discovering efficient ways of coding the sensory data BIB011 . Notably, the application of these algorithms to natural images has been shown to generate receptive field properties similar to those observed in the visual cortex BIB012 . Generative learning can be implemented in the framework of recurrent stochastic neural networks with hidden units  . However, one hidden layer can be insufficient for modeling structured and high-dimensional sensory data. In contrast, a network with many hidden layers, that is a deep network, can learn a more powerful hierarchical generative model BIB017 . Note that a good generative model of the data can be a very useful starting point for later discriminative learning BIB019 BIB023 . The internal representations obtained from generative learning can be the input to a variety of classification or function learning tasks, thereby exploiting re-use of learned features  . Moreover, the internal model might be refined through supervised learning to strengthen the features that are most informative for solving a specific classification task BIB017 ; also see BIB014 , for a related modeling approach to category learning). Indeed, it has been shown that human category learning implies flexibility in the use and creation of perceptual features  and that different types of features might be extracted according to the nature of the learning task (e.g., unsupervised vs. supervised; BIB013 . The goal of the present article is to provide a tutorial overview of generative learning in deep neural networks to highlight its appeal for modeling language and cognition. We start with a brief review of the theoretical foundations of generative learning and deep networks. We then discuss various practical aspects related to training, testing and analyzing deep networks, using the classic letter and word perception problem of  as a tutorial example. The emergence of a hierarchy of orthographic representations through deep unsupervised learning is particularly interesting (also see Di Bono and Zorzi, under review) because it can revisit the hard-wired architecture of the IAM. The idea that perception of written words involves the sensitivity to increasingly larger orthographic units is also supported by recent neuroimaging findings BIB016 BIB020 .