Vector semantics, especially sentence vectors, have recently been used successfully in many areas of natural language processing. However, relatively little work has explored the internal structure and properties of spaces of sentence vectors. In this paper, we will explore the properties of sentence vectors by studying a particular real-world application: Automatic Summarization. In particular, we show that cosine similarity between sentence vectors and document vectors is strongly correlated with sentence importance and that vector semantics can identify and correct gaps between the sentences chosen so far and the document. In addition, we identify specific dimensions which are linked to effective summaries. To our knowledge, this is the first time specific dimensions of sentence embeddings have been connected to sentence properties. We also compare the features of different methods of sentence embeddings. Many of these insights have applications in uses of sentence embeddings far beyond summarization. Vector semantics have been growing in popularity for many other natural language processing applications. Vector semantics attempt to represent words as vectors in a high-dimensional space, where vectors which are close to each other have similar meanings. Various models of vector semantics have been proposed, such as LSA BID10 , word2vec BID14 , and GLOVE BID17 , and these models have proved to be successful in other natural language processing applications.While these models work well for individual words, producing equivalent vectors for sentences or documents has proven to be more difficult.In recent years, a number of techniques for sentence embeddings have emerged. One promising method is paragraph vectors (Also known as Doc2Vec), described by BID12 . The model behind paragraph vectors resembles that behind word2vec, except that a classifier uses an additional 'paragraph vector' to predict words in a Skip-Gram model.Another model, skip-thoughts, attempts to extend the word2vec model in a different way BID9 . The center of the skip-thought model is an encoder-decoder neural network. The result, skip-thought vectors, achieve good performance on a wide variety of natural language tasks.Simpler approaches based on linear combinations of the word vectors have managed to achieve state-of-the-art results for non-domain-specific tasks BID20 . Arora et al. BID1 offer one particularly promising such approach, which was found to achieve equal or greater performance in some tasks than more complicated supervised learning methods. Despite the poor performance of our models compared to the baselines, analyses of the underlying data provide many useful insights into the behavior of vector semantics in real-world tasks. We have identified differences in different forms of sentence vectors when applied to real-world tasks. In particular, each sentence vector form seems to be more successful when used in a particular way. Roughly speaking, Arora's vectors excel at judging the similarity of two sentences while Paragraph Vectors excel at representing document vectors, and at representing features as dimensions of vectors. While we do not have enough data to pinpoint the strengths of Skipthought vectors, they seem to work well in specific contexts that our work did not fully explore. These differences are extremely significant, and will likely make or break real-world applications. Therefore, special care should be taken when selecting the sentence vector method for a real-world task.
A comparison and detailed analysis of various sentence embedding models through the real-world task of automatic summarization.
{"src": "scitldr-AIC_train", "paper_id": "S1347ot3b"}