A Survey of the Usages of Deep Learning in Natural Language Processing <s> Semantic Comparison. <s> We investigate unsupervised techniques for acquiring monolingual sentence-level paraphrases from a corpus of temporally and topically clustered news articles collected from thousands of web-based news sources. Two techniques are employed: (1) simple string edit distance, and (2) a heuristic strategy that pairs initial (presumably summary) sentences from different news stories in the same cluster. We evaluate both datasets using a word alignment algorithm and a metric borrowed from machine translation. Results show that edit distance data is cleaner and more easily-aligned than the heuristic data, with an overall alignment error rate (AER) of 11.58% on a similarly-extracted test set. On test data extracted by the heuristic strategy, however, performance of the two training sets is similar, with AERs of 13.2% and 14.7% respectively. Analysis of 100 pairs of sentences from each set reveals that the edit distance data lacks many of the complex lexical and syntactic alternations that characterize monolingual paraphrase. The summary sentences, while less readily alignable, retain more of the non-trivial alternations that are of greatest interest learning paraphrase relationships. <s> BIB001 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Semantic Comparison. <s> We present a novel approach to deciding whether two sentences hold a paraphrase relationship. We employ a generative model that generates a paraphrase of a given sentence, and we use probabilistic inference to reason about whether two sentences share the paraphrase relationship. The model cleanly incorporates both syntax and lexical semantics using quasi-synchronous dependency grammars (Smith and Eisner, 2006). Furthermore, using a product of experts (Hinton, 2002), we combine the model with a complementary logistic regression model based on state-of-the-art lexical overlap features. We evaluate our models on the task of distinguishing true paraphrase pairs from false ones on a standard corpus, giving competitive state-of-the-art performance. <s> BIB002 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Semantic Comparison. <s> Paraphrase detection is the task of examining two sentences and determining whether they have the same meaning. In order to obtain high accuracy on this task, thorough syntactic and semantic analysis of the two statements is needed. We introduce a method for paraphrase detection based on recursive autoencoders (RAE). Our unsupervised RAEs are based on a novel unfolding objective and learn feature vectors for phrases in syntactic trees. These features are used to measure the word- and phrase-wise similarity between two sentences. Since sentences may be of arbitrary length, the resulting matrix of similarity measures is of variable size. We introduce a novel dynamic pooling layer which computes a fixed-sized representation from the variable-sized matrices. The pooled representation is then used as input to a classifier. Our method outperforms other state-of-the-art approaches on the challenging MSRP paraphrase corpus. <s> BIB003 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Semantic Comparison. <s> Semantic Textual Similarity (STS) measures the degree of semantic equivalence between two texts. This paper presents the results of the STS pilot task in Semeval. The training data contained 2000 sentence pairs from previously existing paraphrase datasets and machine translation evaluation resources. The test data also comprised 2000 sentences pairs for those datasets, plus two surprise datasets with 400 pairs from a different machine translation evaluation corpus and 750 pairs from a lexical resource mapping exercise. The similarity of pairs of sentences was rated on a 0-5 scale (low to high similarity) by human judges using Amazon Mechanical Turk, with high Pearson correlation scores, around 90%. 35 teams participated in the task, submitting 88 runs. The best results scored a Pearson correlation >80%, well above a simple lexical baseline that only scored a 31% correlation. This pilot task opens an exciting way ahead, although there are still open issues, specially the evaluation metric. <s> BIB004 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Semantic Comparison. <s> Large-scale relational learning becomes crucial for handling the huge amounts of structured data generated daily in many application domains ranging from computational biology or information retrieval, to natural language processing. In this paper, we present a new neural network architecture designed to embed multi-relational graphs into a flexible continuous vector space in which the original data is kept and enhanced. The network is trained to encode the semantics of these graphs in order to assign high probabilities to plausible components. We empirically show that it reaches competitive performance in link prediction on standard datasets from the literature as well as on data from a real-world knowledge base (WordNet). In addition, we present how our method can be applied to perform word-sense disambiguation in a context of open-text semantic parsing, where the goal is to learn to assign a structured meaning representation to almost any sentence of free text, demonstrating that it can scale up to tens of thousands of nodes and thousands of types of relation. <s> BIB005 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Semantic Comparison. <s> Semantic word spaces have been very useful but cannot express the meaning of longer phrases in a principled way. Further progress towards understanding compositionality in tasks such as sentiment detection requires richer supervised training and evaluation resources and more powerful models of composition. To remedy this, we introduce a Sentiment Treebank. It includes fine grained sentiment labels for 215,154 phrases in the parse trees of 11,855 sentences and presents new challenges for sentiment compositionality. To address them, we introduce the Recursive Neural Tensor Network. When trained on the new treebank, this model outperforms all previous methods on several metrics. It pushes the state of the art in single sentence positive/negative classification from 80% up to 85.4%. The accuracy of predicting fine-grained sentiment labels for all phrases reaches 80.7%, an improvement of 9.7% over bag of features baselines. Lastly, it is the only model that can accurately capture the effects of negation and its scope at various tree levels for both positive and negative phrases. <s> BIB006 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Semantic Comparison. <s> The ability to accurately represent sentences is central to language understanding. We describe a convolutional architecture dubbed the Dynamic Convolutional Neural Network (DCNN) that we adopt for the semantic modelling of sentences. The network uses Dynamic k-Max Pooling, a global pooling operation over linear sequences. The network handles input sentences of varying length and induces a feature graph over the sentence that is capable of explicitly capturing short and long-range relations. The network does not rely on a parse tree and is easily applicable to any language. We test the DCNN in four experiments: small scale binary and multi-class sentiment prediction, six-way question classification and Twitter sentiment prediction by distant supervision. The network achieves excellent performance in the first three tasks and a greater than 25% error reduction in the last task with respect to the strongest baseline. <s> BIB007 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Semantic Comparison. <s> This paper presents the task on the evaluation of Compositional Distributional Semantics Models on full sentences organized for the first time within SemEval2014. Participation was open to systems based on any approach. Systems were presented with pairs of sentences and were evaluated on their ability to predict human judgments on (i) semantic relatedness and (ii) entailment. The task attracted 21 teams, most of which participated in both subtasks. We received 17 submissions in the relatedness subtask (for a total of 66 runs) and 18 in the entailment subtask (65 runs). <s> BIB008 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Semantic Comparison. <s> In Semantic Textual Similarity, systems rate the degree of semantic equivalence between two text snippets. This year, the participants were challenged with new data sets for English, as well as the introduction of Spanish, as a new language in which to assess semantic similarity. For the English subtask, we exposed the systems to a diversity of testing scenarios, by preparing additional OntoNotesWordNet sense mappings and news headlines, as well as introducing new genres, including image descriptions, DEFT discussion forums, DEFT newswire, and tweet-newswire headline mappings. For Spanish, since, to our knowledge, this is the first time that official evaluations are conducted, we used well-formed text, by featuring sentences extracted from encyclopedic content and newswire. The annotations for both tasks leveraged crowdsourcing. The Spanish subtask engaged 9 teams participating with 22 system runs, and the English subtask attracted 15 teams with 38 system runs. <s> BIB009 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Semantic Comparison. <s> Semantic matching is of central importance to many natural language tasks \cite{bordes2014semantic,RetrievalQA}. A successful matching algorithm needs to adequately model the internal structures of language objects and the interaction between them. As a step toward this goal, we propose convolutional neural network models for matching two sentences, by adapting the convolutional strategy in vision and speech. The proposed models not only nicely represent the hierarchical structures of sentences with their layer-by-layer composition and pooling, but also capture the rich matching patterns at different levels. Our models are rather generic, requiring no prior knowledge on language, and can hence be applied to matching tasks of different nature and in different languages. The empirical study on a variety of matching tasks demonstrates the efficacy of the proposed model on a variety of matching tasks and its superiority to competitor models. <s> BIB010 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Semantic Comparison. <s> Because of their superior ability to preserve sequence information over time, Long Short-Term Memory (LSTM) networks, a type of recurrent neural network with a more complex computational unit, have obtained strong results on a variety of sequence modeling tasks. The only underlying LSTM structure that has been explored so far is a linear chain. However, natural language exhibits syntactic properties that would naturally combine words to phrases. We introduce the Tree-LSTM, a generalization of LSTMs to tree-structured network topologies. Tree-LSTMs outperform all existing systems and strong LSTM baselines on two tasks: predicting the semantic relatedness of two sentences (SemEval 2014, Task 1) and sentiment classification (Stanford Sentiment Treebank). <s> BIB011 </s> A Survey of the Usages of Deep Learning in Natural Language Processing <s> Semantic Comparison. <s> We describe the WIKIQA dataset, a new publicly available set of question and sentence pairs, collected and annotated for research on open-domain question answering. Most previous work on answer sentence selection focuses on a dataset created using the TREC-QA data, which includes editor-generated questions and candidate answer sentences selected by matching content words in the question. WIKIQA is constructed using a more natural process and is more than an order of magnitude larger than the previous dataset. In addition, the WIKIQA dataset also includes questions for which there are no correct sentences, enabling researchers to work on answer triggering, a critical component in any QA system. We compare several systems on the task of answer sentence selection on both datasets and also describe the performance of a system on the problem of answer triggering using the WIKIQA dataset. <s> BIB012
One way to test the efficacy of an approach to computing semantics is to see if two similar phrases, sentences or documents, judged by humans to have similar meaning also are judged similarly by a program. A series of workshops called SemEval have been conducted annually from 1998 to date to encourage research in approaches to semantic computing. Comparing the semantic similarity of two sentences has been a staple contest in SemEval contests, and papers published in other venues. BIB010 proposed two CNNs to perform a semantic comparison task. The first model, ARC-I, inspired by BIB005 , used a "Siamese" network, in which two CNNs sharing weights evaluated two sentences in parallel. In the second network, connections were placed between the two, allowing for sharing before the final states of the CNNs. Three "multiple choice" experiments were performed, testing English sentence completion, responding to Chinese tweets, and comparing English sentence meaning. The approach outperformed a number of existing models. Building on prior work BIB010 BIB007 BIB003 ], Yin and Schütze [2015] proposed a Bi-CNN-MI (MI for multigranular interaction features), consisting of a pretrained CNN sentence model, a CNN interaction model, and a logistic regressor. Following Hu et al., they used a "Siamese" network, however, it was modified by the use of the Dynamic CNNs proposed by Kalchbrenner et al. (described in Section 3.4.2) . Additionally, the feature maps from each level were used in the comparison process, rather than simply the top-level feature maps. They achieved state-of-the-art results on the Microsoft Research Paraphrase Corpus (MSRP) BIB002 BIB001 . also used a CNN to compare the semantic similarity of sentences. The network resembled that of Yin and Schütze, using two CNNs in parallel. The constructed feature maps were then compared using a "similarity measurement layer" followed by a fully-connected layer and then a log-softmax output layer. The windows used in the convolutional layers ranged in length from one to four. The network was trained and evaluated on three different datasets: MSRP, the Sentences Involving Compositional Knowledge (SICK) dataset BIB008 , and the Microsoft Video Paraphrase Corpus (MSRVID) BIB004 . State-of-the-art results were achieved on the first and the third, and nearly matched on the second. Ablation studies showed that a number of elements such as POS tags, multiple pooling operations, and multiple similarity metrics, as well as several other features, all contributed to the performance of the model. The state-of-the-art network that generated superior results to the model proposed by He et al. on the SICK dataset was a model concocted using an RvNN with LSTM-like nodes BIB011 , referred to by its designers as a Tree-LSTM. Two different variations were examined (a constituency-based model and a dependency-based model) and tested on both the SICK dataset and the Stanford Sentiment Treebank BIB006 ]. The constituency-based model achieved state-of-the-art results on the Stanford Sentiment Treebank and the dependency-based model achieved state-of-the-art results on SICK. He et al. presented another model [2016] , which outperformed that of Tai et al. on SICK. The model formed a matrix of the two sentences being compared before applying a "similarity focus layer" and then passing the information into a nineteen-layer CNN followed by dense layers with a softmax output. The similarity focus layer matched semantically similar pairs of words from the input sentences and applied weights to the matrix locations representing the relations between the words in each pair. In addition to testing on the SICK dataset, they tested on MSRVID, SemEval 2014 Task 10 [ BIB009 , WikiQA BIB012 , and TreeQA  ]. State-of-the-art results were attained on all of the above.