A Survey of Machine Learning for Big Code and Naturalness <s> Documentation, Traceability, and Information Retrieval <s> Using the application programming interfaces (API) of large software systems requires developers to understand details about the interfaces that are often not explicitly defined. However, documentation about the API is often incomplete or out of date. Existing systems that make use of the API provide a form of implicit information on how to use that code. Manually searching through existing projects to find relevant source code is tedious and time consuming. We have created the Strathcona Example.Recommendation Tool to assist developers in finding relevant fragments of code, or examples, of an API's use. These examples can be used by developers to provide insight on how they are supposed to interact with the API. <s> BIB001 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Documentation, Traceability, and Information Retrieval <s> Programmers commonly reuse existing frameworks or libraries to reduce software development efforts. One common problem in reusing the existing frameworks or libraries is that the programmers know what type of object that they need, but do not know how to get that object with a specific method sequence. To help programmers to address this issue, we have developed an approach that takes queries of the form "Source object type → Destination object type" as input, and suggests relevant method-invocation sequences that can serve as solutions that yield the destination object from the source object given in the query. Our approach interacts with a code search engine (CSE) to gather relevant code samples and performs static analysis over the gathered samples to extract required sequences. As code samples are collected on demand through CSE, our approach is not limited to queries of any specific set of frameworks or libraries. We have implemented our approach with a tool called PARSEWeb, and conducted four different evaluations to show that our approach is effective in addressing programmer's queries. We also show that PARSEWeb performs better than existing related tools: Prospector and Strathcona <s> BIB002 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Documentation, Traceability, and Information Retrieval <s> Internet-Scale Code Search is the problem of finding source on the Internet. Developers are typically searching for code to reuse as-is on a project or as a reference example. This phenomenon has emerged due to the increasing availability and quality of open source and resources on the web. Solutions to this problem will involve more than the simple application of information retrieval techniques or a scaling-up of tools for code search. Instead, new, purpose-built solutions are needed that draw on results from these areas, as well as program comprehension and software reuse. <s> BIB003 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Documentation, Traceability, and Information Retrieval <s> Key program interfaces are sometimes documented with usage examples: concrete code snippets that characterize common use cases for a particular data type. While such documentation is known to be of great utility, it is burdensome to create and can be incomplete, out of date, or not representative of actual practice. We present an automatic technique for mining and synthesizing succinct and representative human-readable documentation of program interfaces. Our algorithm is based on a combination of path sensitive dataflow analysis, clustering, and pattern abstraction. It produces output in the form of well-typed program snippets which document initialization, method calls, assignments, looping constructs, and exception handling. In a human study involving over 150 participants, 82% of our generated examples were found to be at least as good at human-written instances and 94% were strictly preferred to state of the art code search. <s> BIB004 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Documentation, Traceability, and Information Retrieval <s> Different studies show that programmers are more interested in finding definitions of functions and their uses than variables, statements, or ordinary code fragments. Therefore, developers require support in finding relevant functions and determining how these functions are used. Unfortunately, existing code search engines do not provide enough of this support to developers, thus reducing the effectiveness of code reuse. We provide this support to programmers in a code search system called Portfolio that retrieves and visualizes relevant functions and their usages. We have built Portfolio using a combination of models that address surfing behavior of programmers and sharing related concepts among functions. We conducted two experiments: first, an experiment with 49 C/Cpp programmers to compare Portfolio to Google Code Search and Koders using a standard methodology for evaluating information-retrieval-based engines; and second, an experiment with 19 Java programmers to compare Portfolio to Koders. The results show with strong statistical significance that users find more relevant functions with higher precision with Portfolio than with Google Code Search and Koders. We also show that by using PageRank, Portfolio is able to rank returned relevant functions more efficiently. <s> BIB005 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Documentation, Traceability, and Information Retrieval <s> We present the first method for automatically mining code idioms from a corpus of previously written, idiomatic software projects. We take the view that a code idiom is a syntactic fragment that recurs across projects and has a single semantic purpose. Idioms may have metavariables, such as the body of a for loop. Modern IDEs commonly provide facilities for manually defining idioms and inserting them on demand, but this does not help programmers to write idiomatic code in languages or using libraries with which they are unfamiliar. We present Haggis, a system for mining code idioms that builds on recent advanced techniques from statistical natural language processing, namely, nonparametric Bayesian probabilistic tree substitution grammars. We apply Haggis to several of the most popular open source projects from GitHub. We present a wide range of evidence that the resulting idioms are semantically meaningful, demonstrating that they do indeed recur across software projects and that they occur more frequently in illustrative code examples collected from a Q&A site. Manual examination of the most common idioms indicate that they describe important program concepts, including object creation, exception handling, and resource management. <s> BIB006 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Documentation, Traceability, and Information Retrieval <s> With the advent of large code repositories and sophisticated search capabilities, code search is increasingly becoming a key software development activity. In this work we shed some light into how developers search for code through a case study performed at Google, using a combination of survey and log-analysis methodologies. Our study provides insights into what developers are doing and trying to learn when per- forming a search, search scope, query properties, and what a search session under different contexts usually entails. Our results indicate that programmers search for code very frequently, conducting an average of five search sessions with 12 total queries each workday. The search queries are often targeted at a particular code location and programmers are typically looking for code with which they are somewhat familiar. Further, programmers are generally seeking answers to questions about how to use an API, what code does, why something is failing, or where code is located. <s> BIB007 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Documentation, Traceability, and Information Retrieval <s> Source code examples are used by developers to implement unfamiliar tasks by learning from existing solutions. To better support developers in finding existing solutions, code search engines are designed to locate and rank code examples relevant to user's queries. Essentially, a code search engine provides a ranking schema, which combines a set of ranking features to calculate the relevance between a query and candidate code examples. Consequently, the ranking schema places relevant code examples at the top of the result list. However, it is difficult to determine the configurations of the ranking schemas subjectively. In this paper, we propose a code example search approach that applies a machine learning technique to automatically train a ranking schema. We use the trained ranking schema to rank candidate code examples for new queries at run-time. We evaluate the ranking performance of our approach using a corpus of over 360,000 code snippets crawled from 586 open-source Android projects. The performance evaluation study shows that the learning-to-rank approach can effectively rank code examples, and outperform the existing ranking schemas by about 35.65 % and 48.42 % in terms of normalized discounted cumulative gain (NDCG) and expected reciprocal rank (ERR) measures respectively. <s> BIB008 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Documentation, Traceability, and Information Retrieval <s> We consider the problem of building probabilistic models that jointly model short natural language utterances and source code snippets. The aim is to bring together recent work on statistical modelling of source code and work on bimodal models of images and natural language. The resulting models are useful for a variety of tasks that involve natural language and source code. We demonstrate their performance on two retrieval tasks: retrieving source code snippets given a natural language query, and retrieving natural language descriptions given a source code query (i.e., source code captioning). Experiments show there to be promise in this direction, and that modelling the structure of source code improves performance. <s> BIB009 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Documentation, Traceability, and Information Retrieval <s> Existing API mining algorithms are not yet practical to use as they require expensive parameter tuning and the returned set of API calls can be large, highly redundant and difficult to understand. In an attempt to remedy these shortcomings we present PAM (Probabilistic API Miner), a near parameter-free probabilistic algorithm for mining the most informative API call patterns. We show that PAM significantly outperforms both MAPO and UPMiner, achieving 70% test-set precision, at retrieving relevant API call sequences from GitHub. Moreover, we focus on libraries for which the developers have explicitly provided code examples, yielding over 300,000 LOC of hand-written API example code from the 967 client projects in the data set. This evaluation suggests that the hand-written examples actually have limited coverage of real API usages. <s> BIB010 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Documentation, Traceability, and Information Retrieval <s> Abstract Developers' communication, as contained in emails, issue trackers, and forums, is a precious source of information to support the development process. For example, it can be used to capture knowledge about development practice or about a software project itself. Thus, extracting the content of developers' communication can be useful to support several software engineering tasks, such as program comprehension, source code analysis, and software analytics. However, automating the extraction process is challenging, due to the unstructured nature of free text, which mixes different coding languages (e.g., source code, stack dumps, and log traces) with natural language parts. We conduct an extensive evaluation of Irish (InfoRmation ISlands Hmm), an approach we proposed to extract islands of coded information from free text at token granularity, with respect to the state of art approaches based on island parsing or island parsing combined with machine learners. The evaluation considers a wide set of natural language documents (e.g., textbooks, forum discussions, and development emails) taken from different contexts and encompassing different coding languages. Results indicate an F-measure of Irish between 74% and 99%; this is in line with existing approaches which, differently from Irish , require specific expertise for the definition of regular expressions or grammars. <s> BIB011 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Documentation, Traceability, and Information Retrieval <s> Twitter is one of the most widely used social media platforms today. It enables users to share and view short 140-character messages called “tweets”. About 284 million active users generate close to 500 million tweets per day. Such rapid generation of user generated content in large magnitudes results in the problem of information overload. Users who are interested in information related to a particular domain have limited means to filter out irrelevant tweets and tend to get lost in the huge amount of data they encounter. A recent study by Singer et al. found that software developers use Twitter to stay aware of industry trends, to learn from others, and to network with other developers. However, Singer et al. also reported that developers often find Twitter streams to contain too much noise which is a barrier to the adoption of Twitter. In this paper, to help developers cope with noise, we propose a novel approach named NIRMAL, which automatically identifies software relevant tweets from a collection or stream of tweets. Our approach is based on language modeling which learns a statistical model based on a training corpus (i.e., set of documents). We make use of a subset of posts from StackOverflow, a programming question and answer site, as a training corpus to learn a language model. A corpus of tweets was then used to test the effectiveness of the trained language model. The tweets were sorted based on the rank the model assigned to each of the individual tweets. The top 200 tweets were then manually analyzed to verify whether they are software related or not, and then an accuracy score was calculated. The results show that decent accuracy scores can be achieved by various variants of NIRMAL, which indicates that NIRMAL can effectively identify software related tweets from a huge corpus of tweets. <s> BIB012 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Documentation, Traceability, and Information Retrieval <s> Integrated Development Environments (IDEs) provide a convenient standalone solution that supports developers during various phases of software development. In order to provide better support for developers within such IDEs, we need to understand how much time developers spend using various parts of a given IDE and how often they use available assistance tools. To infer useful conclusions, such information should be gathered for different types of IDEs for different languages. In this paper, we instrument the previously unexplored Visual Studio IDE and track the interactions of developers at an industry partner's software-development department. As a result, we capture interactions for more than 6300 hours of work time, from between 27 and 84 professional C# developers. Our work reports how much time professional developers spend on activities such as code editing and execution or navigation, as well as how often they use assistance tools provided by the IDE. We compare our findings to those of prior studies involving other IDEs and discuss the implications of the commonalities and differences for research on (integrated) developer-assistance tools. <s> BIB013 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Documentation, Traceability, and Information Retrieval <s> Developers often wonder how to implement a certain functionality (e.g., how to parse XML files) using APIs. Obtaining an API usage sequence based on an API-related natural language query is very helpful in this regard. Given a query, existing approaches utilize information retrieval models to search for matching API sequences. These approaches treat queries and APIs as bags-of-words and lack a deep understanding of the semantics of the query. We propose DeepAPI, a deep learning based approach to generate API usage sequences for a given natural language query. Instead of a bag-of-words assumption, it learns the sequence of words in a query and the sequence of associated APIs. DeepAPI adapts a neural language model named RNN Encoder-Decoder. It encodes a word sequence (user query) into a fixed-length context vector, and generates an API sequence based on the context vector. We also augment the RNN Encoder-Decoder by considering the importance of individual APIs. We empirically evaluate our approach with more than 7 million annotated code snippets collected from GitHub. The results show that our approach generates largely accurate API sequences and outperforms the related approaches. <s> BIB014 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Documentation, Traceability, and Information Retrieval <s> Software developers need access to different kinds of information which is often dispersed among different documentation sources, such as API documentation or Stack Overflow. We present an approach to automatically augment API documentation with "insight sentences" from Stack Overflow -- sentences that are related to a particular API type and that provide insight not contained in the API documentation of that type. Based on a development set of 1,574 sentences, we compare the performance of two state-of-the-art summarization techniques as well as a pattern-based approach for insight sentence extraction. We then present SISE, a novel machine learning based approach that uses as features the sentences themselves, their formatting, their question, their answer, and their authors as well as part-of-speech tags and the similarity of a sentence to the corresponding API documentation. With SISE, we were able to achieve a precision of 0.64 and a coverage of 0.7 on the development set. In a comparative study with eight software developers, we found that SISE resulted in the highest number of sentences that were considered to add useful information not found in the API documentation. These results indicate that taking into account the meta data available on Stack Overflow as well as part-of-speech tags can significantly improve unsupervised extraction approaches when applied to Stack Overflow data. <s> BIB015
Improving documentation and code search are central questions in software engineering. Probabilistic models are particularly natural here, because, as we have seen, they allow integrating information between NL text and code. Although the more general code-to-text and text-to-code models from the previous sections could clearly be applied here, researchers have often found, as in the NL domain, that more specialized solutions are currently effective for these problems. Code search-a common activity for software engineers BIB013 BIB007 -can employ natural language queries. Software engineering researchers have focused on the code search problem using information retrieval (IR) methods BIB003 BIB001 BIB005 BIB002 . Niu et al. BIB008 has used learning-to-rank methods but with manually extracted features. Within the area of statistical models of source code, Gu et al. BIB014 train a sequence-to-sequence (seq2seq) neural network to map natural language into API sequences. Allamanis et al. BIB009 learn a bimodal, generative model of code, conditioned on natural language text and use it to rank code search results. All of these methods use rank-based measures (e.g., mean reciprocal rank) to evaluate their performance. Documentation is text that captures requirements, specifications, and descriptions of code. Engineers turn to it to prioritize new features and to understand code during maintenance. Searching, formalizing, reasoning about, and interlinking code to (i.e., the traceability problem of Gotel et al.  ) documentation are seminal software engineering problems. Mining common API patterns is a recurring theme, and there is a large literature of non-probabilistic methods (e.g., frequencybased) for mining and synthesizing API patterns BIB004  , which are out of the scope of this review. Also out of scope is work that combines natural language information with APIs. For example, Treude and Robillard BIB015 extract phrases from StackOverflow using heuristics (manually selected regular expressions) and use off-the-shelf classifiers on a set of hand-crafted features. We refer the reader to Robillard et al.  for all probabilistic and non-probabilistic recommender systems. Within this domain, there are a few probabilistic code models that mine API sequences. Gu et al. BIB014 map natural language text to commonly used API sequences, and Allamanis and Sutton BIB006 learn fine-grained source code idioms that may include APIs. Fowkes and Sutton BIB010 uses a graphical model to mine interesting API sequences. Documentation is also related to information extraction from (potentially unstructured) documents. Cerulo et al. BIB011 use a language model to detect code "islands" in free text. Sharma et al. BIB012 use a language model over tweets to identify software-relevant tweets.