We present the Siamese Continuous Bag of Words ( Siamese CBOW ) model , a neural network for efficient estimation of highquality sentence embeddings . Averaging the embeddings of words in a sentence has proven to be a surprisingly successful and efficient way of obtaining sentence embeddings . However , word embeddings trained with the methods currently available are not optimized for the task of sentence representation , and , thus , likely to be suboptimal . Siamese CBOW handles this problem by training word embeddings directly for the purpose of being averaged . The underlying neural network learns word embeddings by predicting , from a sentence representation , its surrounding sentences . We show the robustness of the Siamese CBOW model by evaluating it on 20 datasets stemming from a wide variety of sources . Word embeddings have proven to be beneficial in a variety of tasks in NLP such as machine translation ( Zou et al . , 2013 ) , parsing ( Chen and Manning , 2014 ) , semantic search ( Reinanda et al . , 2015 ; Voskarides et al . , 2015 ) , and tracking the meaning of words and concepts over time ( Kim et al . , 2014 ; Kenter et al . , 2015 ) . It is not evident , however , how word embeddings should be combined to represent larger pieces of text , like sentences , paragraphs or documents . Surprisingly , simply averaging word embeddings of all words in a text has proven to be a strong baseline or feature across a multitude of tasks ( Faruqui et al . , 2014 ; Yu et al . , 2014 ; Gershman and Tenenbaum , 2015 ; Kenter and de Rijke , 2015 ) . Word embeddings , however , are not optimized specifically for representing sentences . In this paper we present a model for obtaining word embeddings that are tailored specifically for the task of averaging them . We do this by directly including a comparison of sentence embeddings-the averaged embeddings of the words they contain-in the cost function of our network . Word embeddings are typically trained in a fast and scalable way from unlabeled training data . As the training data is unlabeled , word embeddings are usually not task-specific . Rather , word embeddings trained on a large training corpus , like the ones from ( Collobert and Weston , 2008 ; Mikolov et al . , 2013b ) are employed across different tasks ( Socher et al . , 2012 ; Kenter and de Rijke , 2015 ; Hu et al . , 2014 ) . These two qualities-(i ) being trainable from large quantities of unlabeled data in a reasonable amount of time , and ( ii ) robust performance across different tasks-are highly desirable and allow word embeddings to be used in many large-scale applications . In this work we aim to optimize word embeddings for sentence representations in the same manner . We want to produce general purpose sentence embeddings that should score robustly across multiple test sets , and we want to leverage large amounts of unlabeled training material . In the word2vec algorithm , Mikolov et al . ( 2013a ) construe a supervised training criterion for obtaining word embeddings from unsupervised data , by predicting , for every word , its surrounding words . We apply this strategy at the sentence level , where we aim to predict a sentence from its adjacent sentences ( Kiros et al . , 2015 ; Hill et al . , 2016 ) . This allows us to use unlabeled training data , which is easy to obtain ; the only restriction is that documents need to be split into sentences and that the order between sentences is preserved . The main research question we address is whether directly optimizing word embeddings for the task of being averaged to produce sentence embeddings leads to word embeddings that are better suited for this task than word2vec does . Therefore , we test the embeddings in an unsupervised learning scenario . We use 20 evaluation sets that stem from a wide variety of sources ( newswire , video descriptions , dictionary descriptions , microblog posts ) . Furthermore , we analyze the time complexity of our method and compare it to our baselines methods . Summarizing , our main contributions are : We have presented Siamese CBOW , a neural network architecture that efficiently learns word embeddings optimized for producing sentence representations . The model is trained using only unla-beled text data . It predicts , from an input sentence representation , the preceding and following sentence . We evaluated the model on 20 test sets and show that in a majority of cases , 14 out of 20 , Siamese CBOW outperforms a word2vec baseline and a baseline based on the recently proposed skip-thought architecture . As further analysis on various choices of parameters show that the method is stable across settings , we conclude that Siamese CBOW provides a robust way of generating high-quality sentence representations . Word and sentence embeddings are ubiquitous and many different ways of using them in supervised tasks have been proposed . It is beyond the scope of this paper to provide a comprehensive analysis of all supervised methods using word or sentence embeddings and the effect Siamese CBOW would have on them . However , it would be interesting to see how Siamese CBOW embeddings would affect results in supervised tasks . Lastly , although we evaluated Siamese CBOW on sentence pairs , there is no theoretical limitation restricting it to sentences . It would be interesting to see how embeddings for larger pieces of texts , such as documents , would perform in document clustering or filtering tasks .
While an average of word embeddings has proven to be successful as sentence-level representations, it is suboptimal because they are not optimized to represent sentences.
They propose to train word embeddings directly for the purpose of being averaged by predicting sounding sentences from a sentence representation using unlabeled data.
Evaluations show that their word embeddings outperform existing methods in 14 out of 20 datasets and they are stable in choice of parameters.