Vector-space models, from word embeddings to neural network parsers, have many advantages for NLP. But how to generalise from fixed-length word vectors to a vector space for arbitrary linguistic structures is still unclear. In this paper we propose bag-of-vector embeddings of arbitrary linguistic graphs. A bag-of-vector space is the minimal nonparametric extension of a vector space, allowing the representation to grow with the size of the graph, but not tying the representation to any specific tree or graph structure. We propose efficient training and inference algorithms based on tensor factorisation for embedding arbitrary graphs in a bag-of-vector space. We demonstrate the usefulness of this representation by training bag-of-vector embeddings of dependency graphs and evaluating them on unsupervised semantic induction for the Semantic Textual Similarity and Natural Language Inference tasks.
Neural network (NN) models have been used to compute embeddings for parsed sentences. They either use a fixed-length vector for an arbitrarily long sentence (e.g. @cite_15 ), or they keep the original sentence structure and decorate it with vectors (e.g. @cite_21 @cite_3 ). Ours is the first non-parametric vector-based model that does not keep the entire structure. However, attention-based NN models could be used in this way. For example in machine translation, take a source sequence of words and encode it as a sequence of vectors, which are then used in an attention-based NN model to generate the target sequence of words for the translation. Keeping the original ordering of words is not fundamental to this method, and thus it could be interpreted as a bag-of-vector embedding method for sequences. However, it is not at all clear how to generalise such a sequence-encoding method to arbitrary graphs. Similarly, Kalman filters have been used to induce vector representations of word tokens in their sequential context @cite_11 , but it is not clear how to generalise this to arbitrary graphs.
@cite_15 @cite_21 @cite_3 @cite_11
2103305545 2143431851 2133280805
Paraphrase detection is the task of examining two sentences and determining whether they have the same meaning. In order to obtain high accuracy on this task, thorough syntactic and semantic analysis of the two statements is needed. We introduce a method for paraphrase detection based on recursive autoencoders (RAE). Our unsupervised RAEs are based on a novel unfolding objective and learn feature vectors for phrases in syntactic trees. These features are used to measure the word- and phrase-wise similarity between two sentences. Since sentences may be of arbitrary length, the resulting matrix of similarity measures is of variable size. We introduce a novel dynamic pooling layer which computes a fixed-sized representation from the variable-sized matrices. The pooled representation is then used as input to a classifier. Our method outperforms other state-of-the-art approaches on the challenging MSRP paraphrase corpus. We present a neural network method for inducing representations of parse histories and using these history representations to estimate the probabilities needed by a statistical left-corner parser. The resulting statistical parser achieves performance (89.1 F-measure) on the Penn Treebank which is only 0.6 below the best current parser for this task, despite using a smaller vocabulary size and less prior linguistic knowledge. Crucial to this success is the use of structurally determined soft biases in inducing the representation of the parse history, and no use of hard independence assumptions. Natural language parsing has typically been done with small sets of discrete categories such as NP and VP, but this representation does not capture the full syntactic nor semantic richness of linguistic phrases, and attempts to improve on this by lexicalizing phrases or splitting categories only partly address the problem at the cost of huge feature spaces and sparseness. Instead, we introduce a Compositional Vector Grammar (CVG), which combines PCFGs with a syntactically untied recursive neural network that learns syntactico-semantic, compositional vector representations. The CVG improves the PCFG of the Stanford Parser by 3.8 to obtain an F1 score of 90.4 . It is fast to train and implemented approximately as an efficient reranker it is about 20 faster than the current Stanford factored parser. The CVG learns a soft notion of head words and improves performance on the types of ambiguities that require semantic information such as PP attachments.