A Survey of Machine Learning for Big Code and Naturalness <s> Recommender Systems <s> The suggestions made by current IDE's code completion features are based exclusively on static type system of the programming language. As a result, often proposals are made which are irrelevant for a particular working context. Also, these suggestions are ordered alphabetically rather than by their relevance in a particular context. In this paper, we present intelligent code completion systems that learn from existing code repositories. We have implemented three such systems, each using the information contained in repositories in a different way. We perform a large-scale quantitative evaluation of these systems, integrate the best performing one into Eclipse, and evaluate the latter also by a user study. Our experiments give evidence that intelligent code completion systems which learn from examples significantly outperform mainstream code completion systems in terms of the relevance of their suggestions and thus have the potential to enhance developers' productivity. <s> BIB001 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Recommender Systems <s> Natural languages like English are rich, complex, and powerful. The highly creative and graceful use of languages like English and Tamil, by masters like Shakespeare and Avvaiyar, can certainly delight and inspire. But in practice, given cognitive constraints and the exigencies of daily life, most human utterances are far simpler and much more repetitive and predictable. In fact, these utterances can be very usefully modeled using modern statistical methods. This fact has led to the phenomenal success of statistical approaches to speech recognition, natural language translation, question-answering, and text mining and comprehension. We begin with the conjecture that most software is also natural, in the sense that it is created by humans at work, with all the attendant constraints and limitations — and thus, like natural language, it is also likely to be repetitive and predictable. We then proceed to ask whether a) code can be usefully modeled by statistical language models and b) such models can be leveraged to support software engineers. Using the widely adopted n-gram model, we provide empirical evidence supportive of a positive answer to both these questions. We show that code is also very repetitive, and in fact even more so than natural languages. As an example use of the model, we have developed a simple code completion engine for Java that, despite its simplicity, already improves Eclipse's built-in completion capability. We conclude the paper by laying out a vision for future research in this area. <s> BIB002 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Recommender Systems <s> Recent research has successfully applied the statistical n-gram language model to show that source code exhibits a good level of repetition. The n-gram model is shown to have good predictability in supporting code suggestion and completion. However, the state-of-the-art n-gram approach to capture source code regularities/patterns is based only on the lexical information in a local context of the code units. To improve predictability, we introduce SLAMC, a novel statistical semantic language model for source code. It incorporates semantic information into code tokens and models the regularities/patterns of such semantic annotations, called sememes, rather than their lexemes. It combines the local context in semantic n-grams with the global technical concerns/functionality into an n-gram topic model, together with pairwise associations of program elements. Based on SLAMC, we developed a new code suggestion method, which is empirically evaluated on several projects to have relatively 18-68% higher accuracy than the state-of-the-art approach. <s> BIB003 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Recommender Systems <s> Statistical language models have successfully been used to describe and analyze natural language documents. Recent work applying language models to programming languages is focused on the task of predicting code, while mainly ignoring the prediction of programmer comments. In this work, we predict comments from JAVA source files of open source projects, using topic models and n-grams, and we analyze the performance of the models given varying amounts of background data on the project being predicted. We evaluate models on their comment-completion capability in a setting similar to codecompletion tools built into standard code editors, and show that using a comment completion tool can save up to 47% of the comment typing. <s> BIB004 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Recommender Systems <s> Although today’s software systems are composed of a diversity of software artifacts, source code remains the most up-to-date artifact and therefore the most reliable data source. It provides a rich and structured source of information upon which recommendation systems can rely to provide useful recommendations to software developers. Source code-based recommendation systems provide support for tasks such as how to use a given API or framework, provide hints on things missing from the code, suggest how to reuse or correct an existing code, or help novices learn a new project, programming paradigm, language, or style. This chapter highlights relevant decisions involved in developing source code-based recommendation systems. An in-depth presentation of a particular system we developed serves as a concrete illustration of some of the issues that can be encountered and of the development choices that need to be made when building such a system. <s> BIB005 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Recommender Systems <s> The n-gram language model, which has its roots in statistical natural language processing, has been shown to successfully capture the repetitive and predictable regularities (“naturalness") of source code, and help with tasks such as code suggestion, porting, and designing assistive coding devices. However, we show in this paper that this natural-language-based model fails to exploit a special property of source code: localness. We find that human-written programs are localized: they have useful local regularities that can be captured and exploited. We introduce a novel cache language model that consists of both an n-gram and an added “cache" component to exploit localness. We show empirically that the additional cache component greatly improves the n-gram approach by capturing the localness of software, as measured by both cross-entropy and suggestion accuracy. Our model’s suggestion accuracy is actually comparable to a state-of-the-art, semantically augmented language model; but it is simpler and easier to implement. Our cache language model requires nothing beyond lexicalization, and thus is applicable to all programming languages. <s> BIB006 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Recommender Systems <s> We address the problem of synthesizing code completions for programs using APIs. Given a program with holes, we synthesize completions for holes with the most likely sequences of method calls. Our main idea is to reduce the problem of code completion to a natural-language processing problem of predicting probabilities of sentences. We design a simple and scalable static analysis that extracts sequences of method calls from a large codebase, and index these into a statistical language model. We then employ the language model to find the highest ranked sentences, and use them to synthesize a code completion. Our approach is able to synthesize sequences of calls across multiple objects together with their arguments. Experiments show that our approach is fast and effective. Virtually all computed completions typecheck, and the desired completion appears in the top 3 results in 90% of the cases. <s> BIB007 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Recommender Systems <s> We study the problem of building generative models of natural source code (NSC); that is, source code written and understood by humans. Our primary contribution is to describe a family of generative models for NSC that have three key properties: First, they incorporate both sequential and hierarchical structure. Second, we learn a distributed representation of source code elements. Finally, they integrate closely with a compiler, which allows leveraging compiler logic and abstractions when building structure into the model. We also develop an extension that includes more complex structure, refining how the model generates identifier tokens based on what variables are currently in scope. Our models can be learned efficiently, and we show empirically that including appropriate structure greatly improves the models, measured by the probability of generating test programs. <s> BIB008 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Recommender Systems <s> Every programmer has a characteristic style, ranging from preferences about identifier naming to preferences about object relationships and design patterns. Coding conventions define a consistent syntactic style, fostering readability and hence maintainability. When collaborating, programmers strive to obey a project’s coding conventions. However, one third of reviews of changes contain feedback about coding conventions, indicating that programmers do not always follow them and that project members care deeply about adherence. Unfortunately, programmers are often unaware of coding conventions because inferring them requires a global view, one that aggregates the many local decisions programmers make and identifies emergent consensus on style. We present NATURALIZE, a framework that learns the style of a codebase, and suggests revisions to improve stylistic consistency. NATURALIZE builds on recent work in applying statistical natural language processing to source code. We apply NATURALIZE to suggest natural identifier names and formatting conventions. We present four tools focused on ensuring natural code during development and release management, including code review. NATURALIZE achieves 94 % accuracy in its top suggestions for identifier names. We used NATURALIZE to generate 18 patches for 5 open source projects: 14 were accepted. <s> BIB009 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Recommender Systems <s> Code completion is an integral part of modern Integrated Development Environments (IDEs). Developers often use it to explore Application Programming Interfaces (APIs). It is also useful to reduce the required amount of typing and to help avoid typos. Traditional code completion systems propose all type-correct methods to the developer. Such a list is often very long with many irrelevant items. More intelligent code completion systems have been proposed in prior work to reduce the list of proposed methods to relevant items. This work extends one of these existing approaches, the Best Matching Neighbor (BMN) algorithm. We introduce Bayesian networks as an alternative underlying model, use additional context information for more precise recommendations, and apply clustering techniques to improve model sizes. We compare our new approach, Pattern-based Bayesian Networks (PBN), to the existing BMN algorithm. We extend previously used evaluation methodologies and, in addition to prediction quality, we also evaluate model size and inference speed. Our results show that the additional context information we collect improves prediction quality, especially for queries that do not contain method calls. We also show that PBN can obtain comparable prediction quality to BMN, while model size and inference speed scale better with large input sizes. <s> BIB010 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Recommender Systems <s> Nearly every Integrated Development Environment includes a form of code completion. The suggested completions ("suggestions") are typically based on information available at compile time, such as type signatures and variables in scope. A statistical approach, based on estimated models of code patterns in large code corpora, has been demonstrated to be effective at predicting tokens given a context. In this demo, we present CACHECA, an Eclipse plug in that combines the native suggestions with a statistical suggestion regime. We demonstrate that a combination of the two approaches more than doubles Eclipse's suggestion accuracy. A video demonstration is available at https://www.youtube.com/watch?v=3INk0N3JNtc. <s> BIB011 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Recommender Systems <s> Descriptive names are a vital part of readable, and hence maintainable, code. Recent progress on automatically suggesting names for local variables tantalizes with the prospect of replicating that success with method and class names. However, suggesting names for methods and classes is much more difficult. This is because good method and class names need to be functionally descriptive, but suggesting such names requires that the model goes beyond local context. We introduce a neural probabilistic language model for source code that is specifically designed for the method naming problem. Our model learns which names are semantically similar by assigning them to locations, called embeddings, in a high-dimensional continuous space, in such a way that names with similar embeddings tend to be used in similar contexts. These embeddings seem to contain semantic information about tokens, even though they are learned only from statistical co-occurrences of tokens. Furthermore, we introduce a variant of our model that is, to our knowledge, the first that can propose neologisms, names that have not appeared in the training corpus. We obtain state of the art results on the method, class, and even the simpler variable naming tasks. More broadly, the continuous embeddings that are learned by our model have the potential for wide application within software engineering. <s> BIB012 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Recommender Systems <s> Integrated Development Environments (IDEs) provide a convenient standalone solution that supports developers during various phases of software development. In order to provide better support for developers within such IDEs, we need to understand how much time developers spend using various parts of a given IDE and how often they use available assistance tools. To infer useful conclusions, such information should be gathered for different types of IDEs for different languages. In this paper, we instrument the previously unexplored Visual Studio IDE and track the interactions of developers at an industry partner's software-development department. As a result, we capture interactions for more than 6300 hours of work time, from between 27 and 84 professional C# developers. Our work reports how much time professional developers spend on activities such as code editing and execution or navigation, as well as how often they use assistance tools provided by the IDE. We compare our findings to those of prior studies involving other IDEs and discuss the implications of the commonalities and differences for research on (integrated) developer-assistance tools. <s> BIB013 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Recommender Systems <s> We introduce a new generative model for code called probabilistic higher order grammar (PHOG). PHOG generalizes probabilistic context free grammars (PCFGs) by allowing conditioning of a production rule beyond the parent non-terminal, thus capturing rich contexts relevant to programs. Even though PHOG is more powerful than a PCFG, it can be learned from data just as efficiently. We trained a PHOG model on a large JavaScript code corpus and show that it is more precise than existing models, while similarly fast. As a result, PHOG can immediately benefit existing programming tools based on probabilistic models of code. <s> BIB014 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Recommender Systems <s> Attention mechanisms in neural networks have proved useful for problems in which the input and output do not have fixed dimension. Often there exist features that are locally translation invariant and would be valuable for directing the model's attention, but previous attentional architectures are not constructed to learn such features specifically. We introduce an attentional neural network that employs convolution on the input tokens to detect local time-invariant and long-range topical attention features in a context-dependent way. We apply this architecture to the problem of extreme summarization of source code snippets into short, descriptive function name-like summaries. Using those features, the model sequentially generates a summary by marginalizing over two attention mechanisms: one that predicts the next summary token based on the attention weights of the input tokens and another that is able to copy a code token as-is directly into the summary. We demonstrate our convolutional attention neural network's performance on 10 popular Java projects showing that it achieves better performance compared to previous attentional mechanisms. <s> BIB015
Software engineering recommender systems   make recommendations to assist software engineering tasks, such as code autocompletion and recommending likely code reviewers for a given code change. Many of these systems employ data-mining and machine-learning approaches on various software engineering artifacts. Probabilistic models of code find application in source code-based recommender systems BIB005 , such as those that aid developers in writint or maintaining code. Modeling developer intent is a challenge: Even if there were an agreed-on way to formalize intent, developers are reluctant to formalize their intent separately from their code itself. Probabilistic reasoning is well suited for inferring intent, since it allows us to quantify the uncertainty that is inherent to inferring any latent variable. Probabilistic recommender systems extract information from the context of (partial) code and use it to probabilistically reason about developer intent. The most prominent recommender system and a feature commonly used in integrated development environment (IDEs) is code completion. All widely used IDEs, such as Eclipse, IntelliJ, and Visual Studio, have some code completion features. According to Amann et al. BIB013 , code completion is the most used IDE feature. However, code completion tools typically return suggestions in alphabetic order, rather than in relative order of predicted relevance to the context. Statistical code completion aims to improve suggestion accuracy by learning probabilities over the suggestions and providing to the users a ranked list. Some systems focus on automatically completing specific constructs (e.g., method calls and parameters); others try to complete all code tokens. In all cases, probabilistic code completion systems use existing code as their training set. Statistical code completion was first studied by Bruch et al. BIB001 who extracted features from code context to suggest completions for method invocations and constructors. Later, Proksch et al. BIB010 used Bayesian graphical models (structured prediction) to improve accuracy. This contextbased model captures all usages of an object and models the probability distribution for the next call. A version of this research is integrated into the Eclipse IDE under Eclipse Recommenders  . Source code language models have implicitly and explicitly been used for code completion. Hindle et al. BIB002 were the first to use a token-level n-gram LM for this purpose, using the previous n − 1 tokens to represent the completion context at each location. Later, Franks et al. BIB011 and Tu et al. BIB006 used a cache n-gram LM and further improved the completion performance, showing that a local cache acts as a domain adapted n-gram. Nguyen et al. BIB003 augment the completion context with semantic information, improving the code completion accuracy of the n-gram LM. Raychev et al. BIB007 exploit formal properties of the code in context to limit incorrect (but statistically probable) API call suggestions. Their method is the first to depart from simple statistical token completion towards statistical program synthesis of single statements. Apart from tokenlevel language models for code completion, Bielik et al. BIB014 and Maddison and Tarlow BIB008 create AST-level LMs that can be used for suggestion. In contrast to work that predicts source code, Movshovitz-Attias and Cohen BIB004 create a recommender system to assist comment completion given a source code snippet, using a topiclike graphical model to model context information. Similarly, the work of Allamanis et al. BIB009 BIB012 BIB015 can be seen as a recommender systems for suggesting names for variables, methods, and classes by using relevant code tokens as the context.