Representation Learning: A Review and New Perspectives <s> Natural Language Processing (NLP) <s> A goal of statistical language modeling is to learn the joint probability function of sequences of words in a language. This is intrinsically difficult because of the curse of dimensionality: a word sequence on which the model will be tested is likely to be different from all the word sequences seen during training. Traditional but very successful approaches based on n-grams obtain generalization by concatenating very short overlapping sequences seen in the training set. We propose to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences. The model learns simultaneously (1) a distributed representation for each word along with (2) the probability function for word sequences, expressed in terms of these representations. Generalization is obtained because a sequence of words that has never been seen before gets high probability if it is made of words that are similar (in the sense of having a nearby representation) to words forming an already seen sentence. Training such large models (with millions of parameters) within a reasonable time is itself a significant challenge. We report on experiments using neural networks for the probability function, showing on two text corpora that the proposed approach significantly improves on state-of-the-art n-gram models, and that the proposed approach allows to take advantage of longer contexts. <s> BIB001 </s> Representation Learning: A Review and New Perspectives <s> Natural Language Processing (NLP) <s> A dental flossing aid for dispensing and aiding in the use of dental floss for cleaning teeth, comprises an elongated bodymember on which is mounted a supply of dental floss and which comprises a plurality of holes bored through the bodymember. The bodymember is elongated for projecting into the mouth cavity. Dental floss is threaded through the bored holes of the bodymember and extended to the outermost end of the bodymember; the bores serving to develop tension in the dental floss to resist the removal of floss from the floss supply. A pushkey assembly is provided for overcoming the tension and assisting in advancing the dental floss. <s> BIB002 </s> Representation Learning: A Review and New Perspectives <s> Natural Language Processing (NLP) <s> Image annotation datasets are becoming larger and larger, with tens of millions of images and tens of thousands of possible annotations. We propose a strongly performing method that scales to such datasets by simultaneously learning to optimize precision at k of the ranked list of annotations for a given image and learning a low-dimensional joint embedding space for both images and annotations. Our method both outperforms several baseline methods and, in comparison to them, is faster and consumes less memory. We also demonstrate how our method learns an interpretable model, where annotations with alternate spellings or even languages are close in the embedding space. Hence, even when our model does not predict the exact annotation given by a human labeler, it often predicts similar annotations, a fact that we try to quantify by measuring the newly introduced "sibling" precision metric, where our method also obtains excellent results. <s> BIB003 </s> Representation Learning: A Review and New Perspectives <s> Natural Language Processing (NLP) <s> We propose a unified neural network architecture and learning algorithm that can be applied to various natural language processing tasks including part-of-speech tagging, chunking, named entity recognition, and semantic role labeling. This versatility is achieved by trying to avoid task-specific engineering and therefore disregarding a lot of prior knowledge. Instead of exploiting man-made input features carefully optimized for each task, our system learns internal representations on the basis of vast amounts of mostly unlabeled training data. This work is then used as a basis for building a freely available tagging system with good performance and minimal computational requirements. <s> BIB004 </s> Representation Learning: A Review and New Perspectives <s> Natural Language Processing (NLP) <s> We present results obtained with several advanced language modeling techniques, including class based model, cache model, maximum entropy model, structured language model, random forest language model and several types of neural network based language models. We show results obtained after combining all these models by using linear interpolation. We conclude that for both small and moderately sized tasks, we obtain new state of the art results with combination of models, that is significantly better than performance of any individual model. Obtained perplexity reductions against Good-Turing trigram baseline are over 50% and against modified Kneser-Ney smoothed 5-gram over 40%. Index Terms: language modeling, neural networks, model combination, speech recognition <s> BIB005 </s> Representation Learning: A Review and New Perspectives <s> Natural Language Processing (NLP) <s> Paraphrase detection is the task of examining two sentences and determining whether they have the same meaning. In order to obtain high accuracy on this task, thorough syntactic and semantic analysis of the two statements is needed. We introduce a method for paraphrase detection based on recursive autoencoders (RAE). Our unsupervised RAEs are based on a novel unfolding objective and learn feature vectors for phrases in syntactic trees. These features are used to measure the word- and phrase-wise similarity between two sentences. Since sentences may be of arbitrary length, the resulting matrix of similarity measures is of variable size. We introduce a novel dynamic pooling layer which computes a fixed-sized representation from the variable-sized matrices. The pooled representation is then used as input to a classifier. Our method outperforms other state-of-the-art approaches on the challenging MSRP paraphrase corpus. <s> BIB006 </s> Representation Learning: A Review and New Perspectives <s> Natural Language Processing (NLP) <s> The exponential increase in the availability of online reviews and recommendations makes sentiment classification an interesting topic in academic and industrial research. Reviews can span so many different domains that it is difficult to gather annotated training data for all of them. Hence, this paper studies the problem of domain adaptation for sentiment classifiers, hereby a system is trained on labeled reviews from one source domain but is meant to be deployed on another. We propose a deep learning approach which learns to extract a meaningful representation for each review in an unsupervised fashion. Sentiment classifiers trained with this high-level feature representation clearly outperform state-of-the-art methods on a benchmark composed of reviews of 4 types of Amazon products. Furthermore, this method scales well and allowed us to successfully perform domain adaptation on a larger industrial-strength dataset of 22 domains. <s> BIB007 </s> Representation Learning: A Review and New Perspectives <s> Natural Language Processing (NLP) <s> We introduce a novel machine learning framework based on recursive autoencoders for sentence-level prediction of sentiment label distributions. Our method learns vector space representations for multi-word phrases. In sentiment prediction tasks these representations outperform other state-of-the-art approaches on commonly used datasets, such as movie reviews, without using any pre-defined sentiment lexica or polarity shifting rules. We also evaluate the model's ability to predict sentiment distributions on a new dataset based on confessions from the experience project. The dataset consists of personal user stories annotated with multiple labels which, when aggregated, form a multinomial distribution that captures emotional reactions. Our algorithm can more accurately predict distributions over such labels compared to several competitive baselines. <s> BIB008 </s> Representation Learning: A Review and New Perspectives <s> Natural Language Processing (NLP) <s> Language models play an important role in large vocabulary speech recognition and statistical machine translation systems. The dominant approach since several decades are back-off language models. Some years ago, there was a clear tendency to build huge language models trained on hundreds of billions of words. Lately, this tendency has changed and recent works concentrate on data selection. Continuous space methods are a very competitive approach, but they have a high computational complexity and are not yet in widespread use. This paper presents an experimental comparison of all these approaches on a large statistical machine translation task. We also describe an open-source implementation to train and use continuous space language models (CSLM) for such large tasks. We describe an efficient implementation of the CSLM using graphical processing units from Nvidia. By these means, we are able to train an CSLM on more than 500 million words in 20 hours. This CSLM provides an improvement of up to 1.8 BLEU points with respect to the best back-off language model that we were able to build. <s> BIB009 </s> Representation Learning: A Review and New Perspectives <s> Natural Language Processing (NLP) <s> Open-text semantic parsers are designed to interpret any statement in natural language by inferring a corresponding meaning representation (MR – a formal representation of its sense). Unfortunately, large scale systems cannot be easily machine-learned due to a lack of directly supervised data. We propose a method that learns to assign MRs to a wide range of text (using a dictionary of more than 70,000 words mapped to more than 40,000 entities) thanks to a training scheme that combines learning from knowledge bases (e.g. WordNet) with learning from raw text. The model jointly learns representations of words, entities and MRs via a multi-task training process operating on these diverse sources of data. Hence, the system ends up providing methods for knowledge acquisition and wordsense disambiguation within the context of semantic parsing in a single elegant framework. Experiments on these various tasks indicate the promise of the approach. <s> BIB010 </s> Representation Learning: A Review and New Perspectives <s> Natural Language Processing (NLP) <s> This paper extends a novel neural network language model (NNLM) which relies on word clustering to structure the output vocabulary: Structured OUtput Layer (SOUL) NNLM. This model is able to handle arbitrarily-sized vocabularies, hence dispensing with the need for shortlists that are commonly used in NNLMs. Several softmax layers replace the standard output layer in this model. The output structure depends on the word clustering which is based on the continuous word representation determined by the NNLM. Mandarin and Arabic data are used to evaluate the SOUL NNLM accuracy via speech-to-text experiments. Well tuned speech-to-text systems (with error rates around 10%) serve as the baselines. The SOUL model achieves consistent improvements over a classical shortlist NNLM both in terms of perplexity and recognition accuracy for these two languages that are quite different in terms of their internal structure and recognition vocabulary size. An enhanced training scheme is proposed that allows more data to be used at each training iteration of the neural network. <s> BIB011 </s> Representation Learning: A Review and New Perspectives <s> Natural Language Processing (NLP) <s> Data often consists of multiple diverse modalities. For example, images are tagged with textual information and videos are accompanied by audio. Each modality is characterized by having distinct statistical properties. We propose a Deep Boltzmann Machine for learning a generative model of such multimodal data. We show that the model can be used to create fused representations by combining features across modalities. These learned representations are useful for classification and information retrieval. By sampling from the conditional distributions over each data modality, it is possible to create these representations even when some data modalities are missing. We conduct experiments on bimodal image-text and audio-video data. The fused representation achieves good classification results on the MIR-Flickr data set matching or outperforming other deep models as well as SVM based models that use Multiple Kernel Learning. We further demonstrate that this multimodal model helps classification and retrieval even when only unimodal data is available at test time. <s> BIB012
Besides speech recognition, there are many other NLP applications of representation learning. Distributed representations for symbolic data were introduced by Hinton  , and first developed in the context of statistical language modeling by Bengio et al. BIB001 in the so-called neural net language models BIB002 . They are all based on learning a distributed representation for each word, called a word embedding. Adding a convolutional architecture, Collobert et al. BIB004 developed the SENNA system 5 that shares representations across the tasks of language modeling, part-of-speech tagging, chunking, named entity recognition, semantic role labeling, and syntactic parsing. SENNA approaches or surpasses the state of the art on these tasks, but is simpler and much faster than traditional predictors. Learning word embeddings can be combined with learning image representations in a way that allows associating text and images. This approach has been used successfully to build Google's image search, exploiting huge quantities of data to map images and queries in the same space BIB003 , and it has recently been extended to deeper multimodal representations BIB012 . The neural net language model was also improved by adding recurrence to the hidden layers BIB005 , allowing it to beat the state of the art (smoothed n-gram models) not only in terms of perplexity (exponential of the average negative log likelihood of predicting the right next word, going down from 140 to 102), but also in terms of word error rate in speech recognition (since the language model is an important component of a speech recognition system), decreasing it from 17.2 percent (KN5 baseline) or 16.9 percent (discriminative language model) to 14.4 percent on the Wall Street Journal benchmark task. Similar models have been applied in statistical machine translation BIB009 , BIB011 , improving perplexity and BLEU scores. Recursive autoencoders (which generalize recurrent networks) have also been used to beat the state of the art in full sentence paraphrase detection BIB006 , almost doubling the F1 score for paraphrase detection. Representation learning can also be used to perform word sense disambiguation BIB010 , bringing up the accuracy from 67.8 to 70.2 percent on the subset of Senseval-3 where the system could be applied (with subject-verb-object sentences). Finally, it has also been successfully used to surpass the state of the art in sentiment analysis BIB007 , BIB008 .