Despite the fast developmental pace of new sentence embedding methods, it is still challenging to find comprehensive evaluations of these different techniques. In the past years, we saw significant improvements in the field of sentence embeddings and especially towards the development of universal sentence encoders that could provide inductive transfer to a wide variety of downstream tasks. In this work, we perform a comprehensive evaluation of recent methods using a wide variety of downstream and linguistic feature probing tasks. We show that a simple approach using bag-of-words with a recently introduced language model for deep context-dependent word embeddings proved to yield better results in many tasks when compared to sentence encoders trained on entailment datasets. We also show, however, that we are still far away from a universal encoder that can perform consistently across several downstream tasks.
-mean @cite_28 emerged as a response to InferSent @cite_8 and baselines such as Sent2Vec @cite_3 . According to the authors, averaging the word embeddings and comparing with approaches such as InferSent @cite_8 can be unfair due to the difference in embedding dimensions (e.g. 300 vs 4096). -mean is a method that concatenates different word embeddings that represent different information such as syntactic and semantic information, resulting in a larger representation for the word embeddings. In addition, the computation of mean is based on @cite_26 .
@cite_28 @cite_26 @cite_3 @cite_8
2793381978 2251804196 2461583636 2613830894
Average word embeddings are a common baseline for more sophisticated sentence embedding techniques. An important advantage of average word embeddings is their computational and conceptual simplicity. However, they typically fall short of the performances of more complex models such as InferSent. Here, we generalize the concept of average word embeddings to p-mean word embeddings, which are (almost) as efficiently computable. We show that the concatenation of different types of p-mean word embeddings considerably closes the gap to state-of-the-art methods such as InferSent monolingually and substantially outperforms these more complex techniques cross-lingually. In addition, our proposed method outperforms different recently proposed baselines such as SIF and Sent2Vec by a solid margin, thus constituting a much harder-to-beat monolingual baseline for a wide variety of transfer tasks. Our data and code are publicly available. We present a novel, count-based approach to obtaining inter-lingual word representations based on inverted indexing of Wikipedia. We present experiments applying these representations to 17 datasets in document classification, POS tagging, dependency parsing, and word alignment. Our approach has the advantage that it is simple, computationally efficient and almost parameter-free, and, more importantly, it enables multi-source crosslingual learning. In 14 17 cases, we improve over using state-of-the-art bilingual embeddings. Unsupervised word embeddings provide rich linguistic and conceptual information about words. However, they may provide weak information about domain specific semantic relations for certain tasks such as semantic parsing of natural language queries, where such information about words can be valuable. To encode the prior knowledge about the semantic word relations, we present new method as follows: We extend the neural network based lexical word embedding objective function Mikolov, et.al. 2013 by incorporating the information about relationship between entities that we extract from knowledge bases. Our model can jointly learn lexical word representations from free text enriched by the relational word embeddings from relational data (e.g., Freebase) for each type of entity relations. We empirically show on the task of semantic tagging of natural language queries that our enriched embeddings can provide information about not only short-range syntactic dependencies but also long-range semantic dependencies between words. Using the enriched embeddings, we obtain an average of 2 improvement in F-score compared to the previous baselines. XING is a leading career-oriented social networking site in Europe, which usually recommend job ads to their customers. One of the widely used methods in Recomender Systems is content-based filtering, which analyzes the description of item characteristics and the user profile illustrating user's preferences. Due to the sparsity of its dataset, i.e. many job postings are rarely interacted with, XING has been using content-based recommender system to promote the quality of the recommendations. Recent word embedding technique learns semantically meaningful representations for words from co-occurrence in sentences, which enables the effective comparison between words. Based on the Word2Vec technique, XING represents job postings by the average embedding over words they contain. This study explores three alternative methods to represent job postings for the task of recommending jobs to users. In the first experiment, we explore whether the use of a subset of words is more effective to represent the job postings. In the second experiment, instead of averaging over word embeddings, we directly learn document embeddings using Paragraph2Vec. And finally, the third experiment uses Word Mover's Distance to estimate the similarity between job postings. Our experiments show that the embeddings that are learned with Paragraph2Vec result in a better estimation of which job postings are similar, but only when high-dimensional settings are used. The Word Mover's Distance algorithm is computationally expensive, therefore we use existing lower-bounds that allowed us to complete a small-scale experiment within the available time. The results indicate that Word Mover's Distance is not as effective as the average over word embeddings and Paragraph2Vec. In the final part of this thesis, we present the Link2Vec, a novel item representation method based on Word2Vec, which learns semantic representations for items based on the context surrounding the hyperlinks that refer to the item, e.g. hyperlinks to the item's Wikipedia page. Our experiments show that the effectiveness of the embeddings learned with Link2Vec improves with the amount of training data. For the evaluation on the MovieLens dataset, we only obtained a limited set of hyperlinks, which resulted in results that approximate a baseline that uses the average over word embeddings.