Learned representations of source code enable various software developer tools, e.g., to detect bugs or to predict program properties. At the core of code representations often are word embeddings of identifier names in source code, because identifiers account for the majority of source code vocabulary and convey important semantic information. Unfortunately, there currently is no generally accepted way of evaluating the quality of word embeddings of identifiers, and current evaluations are biased toward specific downstream tasks. This paper presents IdBench, the first benchmark for evaluating to what extent word embeddings of identifiers represent semantic relatedness and similarity. The benchmark is based on thousands of ratings gathered by surveying 500 software developers. We use IdBench to evaluate state-of-the-art embedding techniques proposed for natural language, an embedding technique specifically designed for source code, and lexical string distance functions, as these are often used in current developer tools. Our results show that the effectiveness of embeddings varies significantly across different embedding techniques and that the best available embeddings successfully represent semantic relatedness. On the downside, no existing embedding provides a satisfactory representation of semantic similarities, e.g., because embeddings consider identifiers with opposing meanings as similar, which may lead to fatal mistakes in downstream developer tools. IdBench provides a gold standard to guide the development of novel embeddings that address the current limitations.
 Reasoning about source code based on learned representations has various applications, such as predicting method names (Allamanis et al., 2015) , detecting bugs (Pradel & Sen, 2018) and vulnerabilities (Harer et al., 2018) , predicting types (Malik et al., 2019) , detecting similar code (White et al., 2016; Xu et al., 2017) , inferring specifications (DeFreez et al., 2018) , code de-obfuscation (Raychev et al., 2015; Alon et al., 2018a) , and program repair (Devlin et al., 2017) . Many of these techniques are based on embeddings of source code, which map a given piece of code into a continuous vector representation that encodes some aspect of the semantics of the code. A core component of most code embeddings are semantic representations of identifier names, i.e., the names of variables, functions, classes, fields, etc. in source code. Similar to words in natural languages, identifiers are the basic building block of source code. Identifiers not only account for the majority of the vocabulary of source code, but they also convey important information about the (intended) meaning of code. To reason about identifiers and their meaning, code analysis techniques build on learned embeddings of identifiers, either by adapting embeddings that were originally proposed for natural languages (Mikolov et al., 2013a; or with embeddings specifically designed for source code (Alon et al., 2018a) . Given the importance of identifier embeddings, a crucial challenge is measuring how effective an embedding represents the semantic relationships between identifiers. For word embeddings in natural language, the community has addressed this question through a series of gold standards (Finkelstein et al., 2002; Bruni et al., 2014a; Rubenstein & Goodenough, 1965; Miller & Charles, 1991; Hill et al., 2015; Gerz et al., 2016) . These gold standards define how similar two words are based on ratings by human judges, enabling an evaluation that measures how well an embedding reflects the human ratings. Unfortunately, simply reusing existing gold standards to identifiers in source code would be misleading. One reason is that the vocabularies of natural languages and source code overlap only partially, because source code contains various terms and abbreviations not found in natural language texts. Moreover, source code has a constantly growing vocabulary, as developers tend to invent new identifiers, e.g., for newly emerging application domains Babii et al. (2019) . Finally, even words present in both natural languages and source code may differ in their meaning due to computer science-specific meanings of some words, e.g., "float" or "string". This paper addresses the problem of measuring and comparing the effectiveness of embeddings of identifiers. We present IdBench, a benchmark for evaluating techniques that represent semantic similarities of identifiers. The basis of the benchmark is a dataset of developer opinions about the similarity of pairs of identifiers. We gather this dataset through two surveys that show realworld identifiers and code snippets to hundreds of developers, asking them to rate their similarity. Taking the developer opinions as a gold standard, IdBench allows for evaluating embeddings in a systematic way by measuring to what extent an embedding agrees with ratings given by developers. Moreover, inspecting pairs of identifiers for which an embedding strongly agrees or disagrees with the benchmark helps understand the strengths and weaknesses of current embeddings. Overall, we gather thousands of ratings from 500 developers. Cleaning and compiling this raw dataset into a benchmark yields several hundreds of pairs of identifiers with gold standard similarities, including identifiers from a wide range of application domains. We apply our approach to a corpus of JavaScript code, because several recent pieces of work on identifier names and code embeddings focus on this language (Pradel & Sen, 2018; Alon et al., 2018b; a; Malik et al., 2019) . Applying our methodology to another language is straightforward. Based on the newly created benchmark, we evaluate and compare state-of-the-art embeddings of identifiers. We find that different embedding techniques differ heavily in terms of their ability to accurately represent identifier relatedness and similarity. The best available technique, the CBOW variant of FastText, accurately represents relatedness, but none of the available techniques accurately represents identifier similarities. One reason is that some embeddings are confused about identifiers with opposite meaning, e.g., rows and cols, and about identifiers that belong to the same application domain but are not similar. Another reason is that some embeddings miss synonyms, e.g., file and record. We also find that simple string distance functions, which measure the similarity of identifiers without any learning, are surprisingly effective, and even outperform some learned embeddings for the similarity task. In summary, this paper makes the following contributions. (1) Methodology: To the best of our knowledge, we are the first to systematically evaluate embeddings of identifiers. Our methodology is based on surveying developers and summarizing their opinions into gold standard similarities of pairs of identifiers. (2) Reusable benchmark: We make available a benchmark of hundreds of pairs of identifiers, providing a way to systematically evaluate existing and future embeddings. While the best available embeddings are highly effective at representing relatedness, none of the studied techniques reaches the same level of agreement for similarity. In fact, even the best results in Figures 4b and 4c (39%) clearly stay beyond the IRA of our benchmark (62%), showing a huge potential for improvement. For many applications of embeddings of identifiers, semantic similarity is crucial. For example, tools to suggest suitable variable or method names (Allamanis et al., 2015; Alon et al., 2018a) aim for the name that is most similar, not only most related, to the concept represented by the variable or method. Likewise, identifier name-based tools for finding programming errors (Pradel & Sen, 2018) or variable misuses (Allamanis et al., 2017) want to identify situations where the developer uses a wrong, but perhaps related, variable. The lack of embeddings that accurately represent the semantic similarities of identifiers motivates more work on embedding techniques suitable for this task. To better understand why current embeddings sometimes fail to accurately represent similarities, Table 1 shows the most similar identifiers of selected identifiers according to the FastText-cbow and path-based embeddings. The examples illustrate two observations. First, FastText, due to its use of n-grams (Bojanowski et al., 2017) , tends to cluster identifiers based on lexical commonalities. While many lexically similar identifiers are also semantically similar, e.g., substr and substring, this approach misses other synonyms, e.g., item and entry. Another downside is that lexical similarity may also establish wrong relationships. For example, substring and substrCount represent different concepts, but FastText finds them to be highly similar. Second, in contrast to FastText, path-based embeddings tend to cluster words based on their structural and syntactical contexts. This approach helps the embeddings to identify synonyms despite their lexical differences, e.g., count and total, or files and records. The downside is that it also clusters various related but not similar identifiers, e.g., minText and maxText, or substr and getPadding. Some of these identifiers even have opposing meanings, e.g., rows and cols, which can mislead code analysis tools when reasoning about the semantics of code. A somewhat surprising result is that simple string distance functions achieve a level of agreement with IdBench's similarity gold standards as high as some learned embeddings. The reason why string distance functions sometimes correctly identify semantic similarities is that some semantically similar identifiers are also be lexically similar. One downside of lexical approaches is that they miss synonymous identifiers, e.g., count and total. This paper presents the first benchmark for evaluating vector space embeddings of identifiers names, which are at the core of many machine learning models of source code. We compile thousands of ratings gathered from 500 developers into three benchmarks that provide gold standard similarity scores representing the relatedness, similarity, and contextual similarity of identifiers. Using IdBench to experimentally compare five embedding techniques and two string distance functions shows that these techniques differ significantly in their agreement with our gold standard. The best available embedding is very effective at representing how related identifiers are. However, all studied techniques show huge room for improvement in their ability to represent how similar identifiers are. IdBench will help steer future efforts on improved embeddings of identifiers, which will eventually enable better machine learning models of source code.
A benchmark to evaluate neural embeddings of identifiers in source code.
{"src": "scitldr-AIC_train", "paper_id": "SklibJBFDB"}