Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> TESTING A DBN: READ-OUT OF INTERNAL REPRESENTATIONS <s> From the Publisher: ::: This book is a comprehensive introduction to the neural network models currently under intensive study for computational applications. It is a detailed, logically-developed treatment that covers the theory and uses of collective computational networks, including associative memory, feed forward networks, and unsupervised learning. It also provides coverage of neural network applications in a variety of problems of both theoretical and practical interest. <s> BIB001 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> TESTING A DBN: READ-OUT OF INTERNAL REPRESENTATIONS <s> Mounting evidence suggests that ‘core object recognition,' the ability to rapidly recognize objects despite substantial appearance variation, is solved in the brain via a cascade of reflexive, largely feedforward computations that culminate in a powerful neuronal representation in the inferior temporal cortex. However, the algorithm that produces this solution remains poorly understood. Here we review evidence ranging from individual neurons and neuronal populations to behavior and computational models. We propose that understanding this algorithm will require using neuronal and psychophysical data to sift through many computational models, each based on building blocks of small, canonical subnetworks with a common functional goal. <s> BIB002 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> TESTING A DBN: READ-OUT OF INTERNAL REPRESENTATIONS <s> Numerosity estimation is phylogenetically ancient and foundational to human mathematical learning, but its computational bases remain controversial. Here we show that visual numerosity emerges as a statistical property of images in 'deep networks' that learn a hierarchical generative model of the sensory input. Emergent numerosity detectors had response profiles resembling those of monkey parietal neurons and supported numerosity estimation with the same behavioral signature shown by humans and animals. <s> BIB003 </s> Modeling language and cognition with deep unsupervised learning: a tutorial overview <s> TESTING A DBN: READ-OUT OF INTERNAL REPRESENTATIONS <s> Thank you very much for downloading perceptrons an introduction to computational geometry. As you may know, people have search numerous times for their chosen books like this perceptrons an introduction to computational geometry, but end up in malicious downloads. Rather than reading a good book with a cup of coffee in the afternoon, instead they are facing with some harmful virus inside their laptop. <s> BIB004
When performing a discriminative task, one of the simplest methods is to exploit a linear classifier (e.g.,  , to assign a certain class to each input pattern. The classifier makes a decision by using a linear combination of the input features and this represents its main limitation BIB004 . In the case of real sensory signals, this shortcoming is exacerbated by the fact that the feature vectors are high-dimensional and usually lie on highly curved and tangled manifolds BIB002 . However, deep belief networks perform a non-linear projection of the feature vector at each hidden layer, gradually building increasingly more complex and abstract representations of the data that eventually make explicit the latent causes of the sensory signal. This hierarchical organization suggests that a linear "read-out" of hidden unit representations should become increasingly more accurate as a function of layer depth. In this perspective, accuracy of linear read-out can be considered as a coarse measure of how well the relevant features are explicitly encoded at a given depth of the hierarchical generative model (see, e.g., BIB003 ; Di Bono and Zorzi, under review). As noted above, linear read-out can also be used to monitor the quality of the representations developed by the deep network during unsupervised generative learning. The linear read-out on internal representations can be easily implemented using another connectionist module, such as a linear network trained with the delta rule, thereby preserving the biological plausibility of the model. The linear network can also be considered as a response module that supports a particular behavioral task, so that its responses can be assessed against the human data (e.g., numerosity perception in BIB003 , or location-invariant visual word recognition in Di Bono and Zorzi, under review). For example, Stoianov and Zorzi applied this approach to simulate human behavior in a numerosity comparison task after training a DBN on thousands of images of sets of objects. The internal representations at the deepest layer provided the input to a linear network trained to decide whether the numerosity of the input image was larger or smaller than a reference number. Notably, the responses of this decision module were described by a psychometric function that was virtually identical to that of human adults, with the classic modulation by numerical ratio that is the signature of Weber's law for numbers. From a practical point of view, delta rule learning can be conveniently replaced by an equivalent method that is computationally more efficient, which relies on the calculation of a pseudo-inverse matrix BIB001 . Formally, data patterns P = {P 1 , P 2 , . . . , P n } can be associated with desired categories . . , L n } by means of the following linear association: where P and L are matrices containing n column vectors that correspondingly code patterns P i (sensory data or internal representations) and binary class labels L i , and W is the weight matrix of the linear classifier. If an exact solution to this linear system does not exist, a least-mean-square approximation can be found by computing the weight matrix as: where P + is the Moore-Penrose pseudo-inverse  2 As an example, we applied the read-out DBN testing method on the internal representations learned for the images of the four-letter words used in  . We tested two different discriminative problems. The first required the identification of each of the four letters composing a word, using as label a binary vector with one-hot (i.e., localistic) coding of the target letter. The second problem consisted in the identification of the word itself, using as label a binary vector with one-hot coding of the target word. To investigate the quality of the features extracted by deep learning, we compared the classification accuracy on the representations learned at each of the levels of a two-layer DBN (H1 = 120 units, H2 = 200 units) with that of the representations learned by a single RBM with as many hidden units as the top layer of the DBN (H = 200 units). As a baseline, we also measured the classification accuracy obtained by trying to directly categorize the raw input vectors. Note that the read-out of the original data is trivial, due to lack of variability (and noise) in the coding of letters and words (i.e., there is a unique pattern for each letter and word). Indeed, the raw data vectors are linearly separable as shown by the perfect accuracy of the read-out. However, if the input patterns are degraded by adding a certain amount of noise, one should expect a progressive decrease of the classification accuracy when the input representation does not include high-level, invariant features. Indeed, Figure 5 shows that when each word image was corrupted by randomly setting to zero a certain percentage of its pixels, read-out accuracy on the raw pixel data dropped even with a small amount of noise and it approached zero in the word recognition task. As expected, the DBN extracted robust internal representations that were less sensitive to noise. Indeed, both hidden layers supported good discrimination accuracy for letters, whereas only the deepest hidden layer adequately supported word discrimination. Notably, the shallow generative model (RBM) with as many hidden units as the top DBN layer did not unfold word-level information, thereby failing to support robust word recognition (especially for larger noise levels). These results are consistent with the seminal proposal of hierarchical feature processing to 2 In some high-level programming languages, this operation is readily available. For example, in MATLAB/Octave we can use the pinv() function, W = L * pinv(P), or the left matrix divide (a.k.a. "backslash") operator, W = (P \L')'. yield abstract representations of written words  .