A Survey of the Usages of Deep Learning in Natural Language Processing <s> Sentence Modeling. <s> In order to respond correctly to a free form factual question given a large collection of texts, one needs to understand the question to a level that allows determining some of the constraints the question imposes on a possible answer. These constraints may include a semantic classification of the sought after answer and may even suggest using different strategies when looking for and verifying a candidate answer.This paper presents a machine learning approach to question classification. We learn a hierarchical classifier that is guided by a layered semantic hierarchy of answer types, and eventually classifies questions into fine-grained classes. We show accurate results on a large collection of free-form questions used in TREC 10. <s> BIB001 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Sentence Modeling. <s> We examine the task of resolving complex cases of definite pronouns, specifically those for which traditional linguistic constraints on coreference (e.g., Binding Constraints, gender and number agreement) as well as commonly-used resolution heuristics (e.g., string-matching facilities, syntactic salience) are not useful. Being able to solve this task has broader implications in artificial intelligence: a restricted version of it, sometimes referred to as the Winograd Schema Challenge, has been suggested as a conceptually and practically appealing alternative to the Turing Test. We employ a knowledge-rich approach to this task, which yields a pronoun resolver that outperforms state-of-the-art resolvers by nearly 18 points in accuracy on our dataset. <s> BIB002 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Sentence Modeling. <s> Natural language parsing has typically been done with small sets of discrete categories such as NP and VP, but this representation does not capture the full syntactic nor semantic richness of linguistic phrases, and attempts to improve on this by lexicalizing phrases or splitting categories only partly address the problem at the cost of huge feature spaces and sparseness. Instead, we introduce a Compositional Vector Grammar (CVG), which combines PCFGs with a syntactically untied recursive neural network that learns syntactico-semantic, compositional vector representations. The CVG improves the PCFG of the Stanford Parser by 3.8% to obtain an F1 score of 90.4%. It is fast to train and implemented approximately as an efficient reranker it is about 20% faster than the current Stanford factored parser. The CVG learns a soft notion of head words and improves performance on the types of ambiguities that require semantic information such as PP attachments. <s> BIB003 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Sentence Modeling. <s> The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the training speed. By subsampling of the frequent words we obtain significant speedup and also learn more regular word representations. We also describe a simple alternative to the hierarchical softmax called negative sampling. ::: ::: An inherent limitation of word representations is their indifference to word order and their inability to represent idiomatic phrases. For example, the meanings of "Canada" and "Air" cannot be easily combined to obtain "Air Canada". Motivated by this example, we present a simple method for finding phrases in text, and show that learning good vector representations for millions of phrases is possible. <s> BIB004 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Sentence Modeling. <s> Many machine learning algorithms require the input to be represented as a fixed-length feature vector. When it comes to texts, one of the most common fixed-length features is bag-of-words. Despite their popularity, bag-of-words features have two major weaknesses: they lose the ordering of the words and they also ignore semantics of the words. For example, "powerful," "strong" and "Paris" are equally distant. In this paper, we propose Paragraph Vector, an unsupervised algorithm that learns fixed-length feature representations from variable-length pieces of texts, such as sentences, paragraphs, and documents. Our algorithm represents each document by a dense vector which is trained to predict words in the document. Its construction gives our algorithm the potential to overcome the weaknesses of bag-of-words models. Empirical results show that Paragraph Vectors outperforms bag-of-words models as well as other techniques for text representations. Finally, we achieve new state-of-the-art results on several text classification and sentiment analysis tasks. <s> BIB005 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Sentence Modeling. <s> We increase the lexical coverage of FrameNet through automatic paraphrasing. We use crowdsourcing to manually filter out bad paraphrases in order to ensure a high-precision resource. Our expanded FrameNet contains an additional 22K lexical units, a 3-fold increase over the current FrameNet, and achieves 40% better coverage when evaluated in a practical setting on New York Times data. <s> BIB006 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Sentence Modeling. <s> We present the first large-scale, corpus based verification of the seminal work of Dowty on the notion of thematic proto-roles. Our results demonstrate both the need for and the feasibility of a property-based annotation scheme of semantic relationships, as opposed to the currently dominant notion of categorical roles. <s> BIB007 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Sentence Modeling. <s> A fundamental challenge in developing semantic parsers is the paucity of strong supervision in the form of language utterances annotated with logical form. In this paper, we propose to exploit structural regularities in language in different domains, and train semantic parsers over multiple knowledge-bases (KBs), while sharing information across datasets. We find that we can substantially improve parsing accuracy by training a single sequence-to-sequence model over multiple KBs, when providing an encoding of the domain at decoding time. Our model achieves state-of-the-art performance on the Overnight dataset (containing eight domains), improves performance over a single KB baseline from 75.6% to 79.6%, while obtaining a 7x reduction in the number of model parameters. <s> BIB008 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Sentence Modeling. <s> We propose a process for investigating the extent to which sentence representations arising from neural machine translation (NMT) systems encode distinct semantic phenomena. We use these representations as features to train a natural language inference (NLI) classifier based on datasets recast from existing semantic annotations. In applying this process to a representative NMT system, we find its encoder appears most suited to supporting inferences at the syntax-semantics interface, as compared to anaphora resolution requiring world-knowledge. We conclude with a discussion on the merits and potential deficiencies of the existing process, and how it may be improved and extended as a broader framework for evaluating semantic coverage. <s> BIB009 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Sentence Modeling. <s> We propose a hypothesis only baseline for diagnosing Natural Language Inference (NLI). Especially when an NLI dataset assumes inference is occurring based purely on the relationship between a context and a hypothesis, it follows that assessing entailment relations while ignoring the provided context is a degenerate solution. Yet, through experiments on ten distinct NLI datasets, we find that this approach, which we refer to as a hypothesis-only model, is able to significantly outperform a majority class baseline across a number of NLI datasets. Our analysis suggests that statistical irregularities may allow a model to perform NLI in some datasets beyond what should be achievable without access to the context. <s> BIB010 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Sentence Modeling. <s> We train multi-task autoencoders on linguistic tasks and analyze the learned hidden sentence representations. The representations change significantly when translation and part-of-speech decoders are added. The more decoders a model employs, the better it clusters sentences according to their syntactic similarity, as the representation space becomes less entangled. We explore the structure of the representation space by interpolating between sentences, which yields interesting pseudo-English sentences, many of which have recognizable syntactic structure. Lastly, we point out an interesting property of our models: The difference-vector between two sentences can be added to change a third sentence with similar features in a meaningful way. <s> BIB011
While the comparison of the meanings of sentences and phrases is useful, it in large part relies on being able to calculate the meanings of individual sentences and phrases. This is a common theme in all of NLP. Extending from neural language modeling, which attempts to capture the meaning of words in vectors, sentence modeling attempts to capture the meaning of sentences in vectors. Taking this a step further are models, such as that of BIB005 , which attempt to model paragraphs or larger bodies of text in this way. While these are intriguing, there is still much work to be done on the phrase and sentence levels, which will be the focus of the remainder of this section. Kalchbrenner et al. [2014] generated representations of sentences using a dynamic convolutional neural network (DCNN), which used a number of different filters and dynamic k-max pooling layers to determine which features detected by those filters are most important. Due to the dynamic pooling, features of different types and lengths could be identified in sentences with varying structures without padding of the input. This allowed not only short-range dependencies, but also long-range dependencies to be identified. The DCNN was tested in several applied tasks that require thorough semantic understanding. It outperformed all comparison models in predicting sentiment of movie reviews in the Stanford Sentiment Treebank BIB003 and in identification of sentiment in tweets  ]. It was also one of the top performers in classifying types of questions using the TREC database BIB001 . Undoubtedly, the area of NLP space most dependent on good semantic understanding is that of machine translation. Between their requirement for such understanding and their ease of examination due to the typical encoder-decoder structure they use, neural machine translation (NMT) systems are splendid testbeds for researching internal semantic representations. One study that employed this, the results of which were published during the writing of this survey, was that conducted by BIB009 . Encoders were trained on four different language pairs: English and Arabic, English and Spanish, English and Chinese, and English and German. Building on a number of previous studies, the decoding classifiers were trained on four distinct datasets: Multi-NLI , which is an expanded version of SNLI , as well as three recast datasets from the JHU Decompositional Semantics Initiative [White et al. [n. d.] ] (FrameNet Plus or FN+ BIB006 , Definite Pronoun Resolution or DPR BIB002 , and Semantic Proto-Roles or SPR BIB007 ). None of the results were particularly strong, although they were strongest in SPR. This led the researchers to the conclusion that NMT models do a poor job of capturing paraphrased information and fail to capture inferences that help in anaphora resolution (e.g. resolving gender, plurality, etc.). They did, however, find that the models may learn a reasonable amount about proto-roles (e.g. who or what is the recipient of an action). A concurrent work BIB010 ] analyzed the quality of many datasets used for natural language inference. BIB008 found that training semantic parsers on a single domain, as is often done, is less effective than training across many domains. This conclusion was drawn after constructing and testing three different LSTM-based models. The first model was a one-to-one model, in which a single encoder and single decoder were used, requiring the network itself to determine the domain of the input. In the second model, a many-to-many model, a decoder was used for each domain, as were two encoders: the domain specific encoder, and a multidomain encoder. The third model was a one-to-many model, using a single encoder, but separate decoders for each domain. Each model was trained on the "OVERNIGHT" dataset . Exceptional results were achieved for all models, with a state-of-the-art performance exhibited by the one-to-one model. Similar conclusions were drawn in a recently published study by BIB011 . This study also has some of the most significant implications for the entire field of natural language processing since Mikolov's arithmetic with word embeddings BIB004 . In this study, several LSTM-based encoder-decoder networks were created, and the embedding vectors produced were analyzed. A single encoder accepting English sentences as input was used, as were four different decoders. The first such decoder was a replicating decoder, which attempted to reproduce the original English input. The second and third decoders attempted to translate the text into German and French. Finally, the fourth decoder was a POS tagger. Different combinations of decoders were used; one model had only the replicating decoder while others had two, three, or, in one case, all four. Sentences of fourteen different structures from the EuroParl dataset  were used to train the networks. A set of test sentences were then fed to the encoders and their output analyzed. In all cases, fourteen clusters were formed, each corresponding to one of the sentence structures. Analyzing the spacing between the clusters and the number of misclassified results showed that adding more decoders led to more correct and more definitive clusters. In particular, using all four of the decoders led to a zero error rate. Furthermore, the researchers tested and confirmed a hypothesis regarding these sentence embeddings. They found that just as logical arithmetic can be performed on word embeddings, so can it be performed on sentence embeddings. For example, taking the vector embedding for, "This example works, " and subtracting the vector for, "This example fails, " produced a vector which when added to the embedding for, "Another attempt fails, " resulted in a vector nearly identical to the embedding of, "Another attempts work. " Note that while this sentence is incorrect, it is very close to the desired result. Furthermore, it differs from the desired sentence only in terms of anaphora, which was noted above to be the most difficult semantic construct to process. While this is acknowledged to be a well-controlled environment, it shows that NLP models are capable of capturing a great deal of meaning and paves the way for a new line of research, and potentially another revolution in the field.