Prepositions are among the most frequent words. Good prepositional representation  is of great syntactic and semantic interest  in computational linguistics. Existing methods on preposition representation either treat prepositions as content words (e.g., word2vec and GloVe) or depend heavily on external linguistic resources including syntactic parsing, training task and dataset-specific representations. In this paper we use word-triple counts (one of the words is a preposition) to  capture the preposition's interaction with its head and children. Prepositional  embeddings are derived via tensor decompositions on a large unlabeled corpus.   We reveal a new geometry involving Hadamard products and empirically demonstrate its utility in paraphrasing of phrasal verbs. Furthermore, our prepositional  embeddings are used as simple features to two challenging downstream tasks: preposition selection and prepositional attachment disambiguation. We achieve comparable to or better results than state of the art on  multiple standardized datasets.   Prepositions are a linguistically closed class comprising some of the most frequent words; they play an important role in the English language since they encode rich syntactic and semantic information. Many preposition-related tasks still remain unsolved in computational linguistics because of their polysemous nature and flexible usage patterns. An accurate understanding and representation of prepositions' linguistic role is key to several important NLP tasks such as grammatical error correction and prepositional phrase attachment. A first-order approach is to represent prepositions as real-valued vectors via word embeddings such as word2vec BID21 and GloVe BID25 .Word embeddings have brought a renaissance in NLP research; they have been very successful in capturing word similarities as well as analogies (both syntactic and semantic) and are now mainstream in nearly all downstream NLP tasks (such as question-answering). Despite this success, no specific properties of word embeddings of prepositions have been highlighted in the literature. Indeed, many of the common prepositions have very similar vector representations as shown in TAB0 for preposition vectors trained using word2vec and GloVe.While this suggests that using available representations for prepositions diminishes the distinguishing feature between prepositions, one could hypothesize that this is primarily because standard word embedding algorithms treat prepositions no different from other content words such as verbs and nouns, i.e., embeddings are created based on co-occurrences with other words. However , prepositions are very frequent and co-occur with nearly all words, which means that their co-occurrence ought to be treated differently. Modern descriptive linguistic theory proposes to understand a preposition via its interactions with both the head (attachment) and child (complement) BID12 ; BID8 . This theory naturally suggests that one should count co-occurrences of a given preposition with pairs of neighboring words. One way of achieving this would be by considering a tensor of triples (word 1 , word 2 , preposition), where we do not restrict word 1 and word 2 to be head-and child-words; instead we model a preposition's interaction with all pairs of neighboring words via a slice of a tensor X -the slice is populated by word co-occurrences restricted to a context window of the specific preposition. Thus, the tensor dimension is V × V × K where V is the vocabulary and K is the number of prepositions; since K ≈ 50, we note that V K.Using such a representation, we find that the resulting tensor is low rank and extract embeddings for both preposition and non-preposition words using a combination of standard ideas from word representations (such as weighted spectral decomposition as in GloVe BID25 ) and tensor decompositions (alternating least squares (ALS) methods BID29 ). The preposition embeddings are discriminative, see preposition similarity of the tensor embedding in TAB0 . We demonstrate that the resulting representation for prepositions captures the core linguistic property of prepositions. We do this using both intrinsic evaluations and downstream tasks, where we provide new state-of-the-art results on well-known NLP tasks involving prepositions.Intrinsic evaluations: We show that the Hadamard product of the embeddings of a verb and a preposition closely approximates the representation of this phrasal verb's paraphrase. Example: v made v from ≈ v produced where represents the Hadamard product (i.e., pointwise multiplication) of two vectors; this approximation does not hold for the standard word embeddings of prepositions (word2vec or GloVe). We provide a mathematical interpretation for this new geometry as well as empirically demonstrate the generalization on a new data set of compositional phrasal verbs.Extrinsic evaluations: Our preposition embeddings are used as features for a simple classifier in two well-known challenging downstream NLP classification tasks. In both tasks, we perform comparable to or strictly better than the state-of-the-art on multiple standardized datasets.Preposition selection: The choice of prepositions significantly influences (and is governed by) the semantics of the context they occur in. Furthermore, the prepositional choice is usually very subtle (and consequently is one of the most frequent error types made by second language English speakers BID19 ). This task tests the choice of a preposition in a large set of contexts (7, 000 instances of both CoNLL-2013 and SE datasets BID26 ). Our approach achieves 6% and 2% absolute improvement over the previous state-of-the-art on the respective datasets.Prepositional attachment disambiguation: Prepositional phrase attachment is a common cause of structural ambiguity in natural language. In the sentence "Pierre Vinken joined the board as a voting member", the prepositional phrase "as a voting member" can attach to either "joined" (the VP) or "the board" (the NP); in this case the VP attachment is correct. Despite extensive study over decades of research, prepositional attachment continues to be a major source of syntactic parsing errors BID4 ; Kummerfeld et al. (2012); BID7 . We use our prepositional representations as simple features to a standard classifier on this task. Our approach tested on a widely studied standard dataset BID2 achieves 89% accuracy, essentially the same performance as state-of-the-art (90% accuracy). It is noteworthy that while the stateof-the-art results are obtained with significant linguistic resources, including syntactic parsers and WordNet, our approach does not rely on these resources to achieve a comparable performance.We emphasize two aspects of our contributions:(1) It is folklore within the NLP community that representations via pairwise word counts capture much of the benefits of the unlabeled sentence-data; example: BID29 reports that their word representations via word-triple counts are better than others, but still significantly worse than regular word2vec representations. One of our main observations is that considering word-triple counts makes most (linguistic) sense when one of the words is a preposition. Furthermore, the sparsity of the corresponding tensor is no worse than the sparsity of the regular word co-occurrence matrix (since prepositions are so frequent and co-occur with essentially every word). Taken together, these two points strongly suggest the benefits of tensor representations in the context of prepositions.(2) The word and preposition representations via tensor decomposition are simple features leading to a standard classifier. In particular, we do not use syntactic parsing (which many prior methods have relied on) or handcrafted features BID26 or train task-specific representations on the annotated training dataset BID2 . The simplicity combined with our strong empirical results (new state-of-the-art results on long standing datasets) lends credence to the strength of the prepositional representations found via tensor decompositions. Co-occurrence counts of word pairs in sentences and the resulting word vector representations (embeddings) have revolutionalized NLP research. A natural generalization is to consider co-occurrence counts of word triples, resulting in a third order tensor. Partly due to the size of the tensor (a vocabulary of 1M, leads to a tensor with 10 18 entries!) and partly due to the extreme dynamic range of entries (including sparsity), word vector representations via tensor decompositions have largely been inferior to their lower order cousins (i.e., regular word embeddings).In this work, we trek this well-trodden terrain but restricting word triples to the scenario when one of the words is a preposition. This is linguistically justified, since prepositions are understood to model interactions between pairs of words. Numerically , this is also very well justified since the sparsity and dynamic range of the resulting tensor is no worse than the original matrix of pairwise co-occurrence counts; this is because prepositions are very frequent and co-occur with essentially every word in the vocabulary.Our intrinsic evaluations and new state of the art results in downstream evaluations lend strong credence to the tensor-based approach to prepositional representation. We expect our vector representations of prepositions to be widely used in more complicated downstream NLP tasks where prepositional role is crucial, including "text to programs" BID10 .
This work is about tensor-based method for preposition representation training.
{"src": "scitldr-AIC_train", "paper_id": "HyB9Np6WG"}