Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> CONNECTIONIST MODELING WITH DEEP NETWORKS: A TUTORIAL <s> Multilayer Neural Networks trained with the backpropagation algorithm constitute the best example of a successful Gradient-Based Learning technique. Given an appropriate network architecture, Gradient-Based Learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional Neural Networks, that are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation, recognition, and language modeling. A new learning paradigm, called Graph Transformer Networks (GTN), allows such multi-module systems to be trained globally using Gradient-Based methods so as to minimize an overall performance measure. Two systems for on-line handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of Graph Transformer Networks. A Graph Transformer Network for reading bank check is also described. It uses Convolutional Neural Network character recognizers combined with global training techniques to provides record accuracy on business and personal checks. It is deployed commercially and reads several million checks per day. <s> BIB001 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> CONNECTIONIST MODELING WITH DEEP NETWORKS: A TUTORIAL <s> Matching Exact Posterior Probabilities in the Multinomial Interactive Activation Model Pranav Khaitan Stanford University James L. McClelland Stanford University Abstract: Interactive activation models of context effects in perception ((McClelland & Rumelhart, 1981; McClel- land & Elman, 1986) have been criticized for failing to combine stimulus and context information in a Bayes-optimal way, leading to a rejection of interactive approaches (Norris & McQueen, 2008). We show that interactive activation can compute correct Bayesian posterior probabilities. We present a variant of the interactive activation model that produces outputs exactly corresponding to the correct posterior probabilities of letters given specified letter feature and context information. In the new variant of the model, inhibition between units within pools is replaced by selection of a single unit to be active, using the softmax function to assign probabilities to candidate alternatives. The model is fully interactive, yet the probability of a letter unit being activated is both provably and demonstrably equal to the posterior probability given the presented feature and context information. <s> BIB002 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> CONNECTIONIST MODELING WITH DEEP NETWORKS: A TUTORIAL <s> We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overriding in the fully-connected layers we employed a recently-developed regularization method called "dropout" that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry. <s> BIB003
In this section we provide a practical overview on how to construct a complete DBN simulation. We illustrate how to train, test and analyze a deep network model using the classic letter and word perception problem of  . Written word perception is particularly representative because it can be linked to one of the most influential models of language processing, McClelland and Rumehart's IAM, and more specifically to its two key assumptions: (1) a hierarchical organization of the network, with increasingly more complex levels of representation, and (2) the mixing of bottom-up and top-down processing (i.e., interactivity) to resolve ambiguity of the sensory input. Interestingly, a recent re-formulation of the IAM as a probabilistic generative model BIB002 ) was shown to perform optimal Bayesian inference, thereby supporting the appeal of the hierarchical interactive architecture (Mirman et al., in press) . A deep learning model would therefore represent an important step forward, because the hard-wired architecture of the IAM might be replaced by the hierarchical generative model learned in a DBN. In this regard, learning word perception can be seen as a stochastic inference problem where the goal is to estimate the posterior distribution over latent variables given the image of a word as input. Though written word perception is an excellent candidate for deep learning, the complexity of the problem makes realistic simulations difficult to handle. For example, high-resolution images of whole words would require a very large network, with tens of thousands of visible units (e.g., 20,000 units for a 400 by 50 pixels image), many hidden layers and billions of connections (see BIB003 , for deep learning on a realistic object recognition problem). One possible simplification would be to split words into letter constituents and first model the perception of single letters. This might lead to sensible internal letter representations that are invariant to position, size, rotation, and noise (i.e., abstract letter identities;  . Alternatively, written words can be represented using small resolution images, with letters encoded as combinations of simple geometric features (the "Siple" font;  . We employed the latter solution for the simulations presented here. In this tutorial we also consider deep learning of handwritten digits (MNIST database; BIB001 and visual numerosity estimation in relation to the analysis of DBNs, because they represent more realistic perception problems that involve training on thousands of images. Training on a large dataset can be important for the emergence of a richer hierarchical structure of features.