As distributed approaches to natural language semantics have developed and diversified, embedders for linguistic units larger than words (e.g., sentences) have come to play an increasingly important role.   To date, such embedders have been evaluated using benchmark tasks (e.g., GLUE) and linguistic probes.   We propose a comparative approach, nearest neighbor overlap (N2O), that quantifies similarity between embedders in a task-agnostic manner.   N2O requires only a collection of examples and is simple to understand: two embedders are more similar if, for the same set of inputs, there is greater overlap between the inputs' nearest neighbors.   We use N2O to compare 21 sentence embedders and show the effects of different design choices and architectures. Continuous embeddings-of words and of larger linguistic units-are now ubiquitious in NLP. The success of self-supervised pretraining methods that deliver embeddings from raw corpora has led to a proliferation of embedding methods, with an eye toward "universality" across NLP tasks. Our focus here is on sentence embedders, and specifically their evaluation. As with most NLP components, intrinsic (e.g., and extrinsic (e.g., GLUE; Wang et al., 2019) evaluations have emerged for sentence embedders. Our approach, nearest neighbor overlap (N2O), is different: it compares a pair of embedders in a linguistics-and task-agnostic manner, using only a large unannotated corpus. The central idea is that two embedders are more similar if, for a fixed query sentence, they tend to find nearest neighbor sets that overlap to a large degree. By drawing a random sample of queries from the corpus itself, N2O can be computed on in-domain data without additional annotation, and therefore can help inform embedder choices in applications such as text clustering (Cutting et al., 1992) , information retrieval (Salton & Buckley, 1988) , and open-domain question answering (Seo et al., 2018) , among others. After motivating and explaining the N2O method ( §2), we apply it to 21 sentence embedders ( §3-4). Our findings ( §5) reveal relatively high functional similarity among averaged static (noncontextual) word type embeddings, a strong effect of the use of subword information, and that BERT and GPT are distant outliers. In §6, we demonstrate the robustness of N2O across different query samples and probe sizes. We also illustrate additional analyses made possible by N2O: identifying embeddingspace neighbors of a query sentence that are stable across embedders, and those that are not ( §7); and probing the abilities of embedders to find a known paraphrase ( §8). The latter reveals considerable variance across embedders' ability to identify semantically similar sentences from a broader corpus. In this paper, we introduce nearest neighbor overlap (N2O), a comparative approach to quantifying similarity between sentence embedders. Using N2O, we draw comparisons across 21 embedders. We also provide additional analyses made possible with N2O, from which we find high variation in embedders' treatment of semantic similarity. GloVe. We use three sets of standard pretrained GloVe embeddings: 100D and 300D embeddings trained on Wikipedia and Gigaword (6B tokens), and 300D embeddings trained on Common Crawl (840B tokens). 13 We handle tokenization and embedding lookup identically to word2vec; for the Wikipedia/Gigaword embeddings, which are uncased, we lower case all tokens as well. FastText. We use four sets of pretrained FastText embeddings: two trained on Wikipedia and other news corpora, and two trained on Common Crawl (each with an original version and one trained on subword information). 14 We use the Python port of the FastText implementation to handle tokenization, embedding lookup, and OOV embedding computation.
We propose nearest neighbor overlap, a procedure which quantifies similarity between embedders in a task-agnostic manner, and use it to compare 21 sentence embedders.
{"src": "scitldr-AIC_train", "paper_id": "ByePEC4KDS"}