A Survey of Machine Learning for Big Code and Naturalness <s> Code to Text and Text to Code <s> Statistical language models have successfully been used to describe and analyze natural language documents. Recent work applying language models to programming languages is focused on the task of predicting code, while mainly ignoring the prediction of programmer comments. In this work, we predict comments from JAVA source files of open source projects, using topic models and n-grams, and we analyze the performance of the models given varying amounts of background data on the project being predicted. We evaluate models on their comment-completion capability in a setting similar to codecompletion tools built into standard code editors, and show that using a comment completion tool can save up to 47% of the comment typing. <s> BIB001 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Code to Text and Text to Code <s> We consider the problem of translating natural language text queries into regular expressions which represent their meaning. The mismatch in the level of abstraction between the natural language representation and the regular expression representation make this a novel and challenging problem. However, a given regular expression can be written in many semantically equivalent forms, and we exploit this flexibility to facilitate translation by finding a form which more directly corresponds to the natural language. We evaluate our technique on a set of natural language queries and their associated regular expressions which we gathered from Amazon Mechanical Turk. Our model substantially outperforms a stateof-the-art semantic parsing baseline, yielding a 29% absolute improvement in accuracy. 1 <s> BIB002 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Code to Text and Text to Code <s> Millions of computer end users need to perform tasks over tabular spreadsheet data, yet lack the programming knowledge to do such tasks automatically. This paper describes the design and implementation of a robust natural language based interface to spreadsheet programming. Our methodology involves designing a typed domain-specific language (DSL) that supports an expressive algebra of map, filter, reduce, join, and formatting capabilities at a level of abstraction appropriate for non-expert users. The key algorithmic component of our methodology is a translation algorithm for converting a natural language specification in the context of a given spreadsheet to a ranked set of likely programs in the DSL. The translation algorithm leverages the spreadsheet spatial and temporal context to assign interpretations to specifications with implicit references, and is thus robust to a variety of ways in which end users can express the same task. The translation algorithm builds over ideas from keyword programming and semantic parsing to achieve both high precision and high recall. We implemented the system as an Excel add-in called NLyze that supports a rich user interaction model including annotating the user's natural language specification and explaining the synthesized DSL programs by paraphrasing them into structured English. We collected a total of 3570 English descriptions for 40 spreadsheet tasks and our system was able to generate the intended interpretation as the top candidate for 94% (97 for the top 3) of those instances. <s> BIB003 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Code to Text and Text to Code <s> Pseudo-code written in natural language can aid the comprehension of source code in unfamiliar programming languages. However, the great majority of source code has no corresponding pseudo-code, because pseudo-code is redundant and laborious to create. If pseudo-code could be generated automatically and instantly from given source code, we could allow for on-demand production of pseudo-code without human effort. In this paper, we propose a method to automatically generate pseudo-code from source code, specifically adopting the statistical machine translation (SMT) framework. SMT, which was originally designed to translate between two natural languages, allows us to automatically learn the relationship between source code/pseudo-code pairs, making it possible to create a pseudo-code generator with less human effort. In experiments, we generated English or Japanese pseudo-code from Python statements using SMT, and find that the generated pseudo-code is largely accurate, and aids code understanding. <s> BIB004 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Code to Text and Text to Code <s> We present a new code assistance tool for integrated development environments. Our system accepts as input free-form queries containing a mixture of English and Java, and produces Java code expressions that take the query into account and respect syntax, types, and scoping rules of Java, as well as statistical usage patterns. In contrast to solutions based on code search, the results returned by our tool need not directly correspond to any previously seen code fragment. As part of our system we have constructed a probabilistic context free grammar for Java constructs and library invocations, as well as an algorithm that uses a customized natural language processing tool chain to extract information from free-form text queries. We present the results on a number of examples showing that our technique (1) often produces the expected code fragments, (2) tolerates much of the flexibility of natural language, and (3) can repair incorrect Java expressions that use, for example, the wrong syntax or missing arguments. <s> BIB005 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Code to Text and Text to Code <s> Using natural language to write programs is a touchstone problem for computational linguistics. We present an approach that learns to map natural-language descriptions of simple “if-then” rules to executable code. By training and testing on a large corpus of naturally-occurring programs (called “recipes”) and their natural language descriptions, we demonstrate the ability to effectively map language to code. We compare a number of semantic parsing approaches on the highly noisy training data collected from ordinary users, and find that loosely synchronous systems perform best. <s> BIB006 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Code to Text and Text to Code <s> High quality source code is often paired with high level summaries of the computation it performs, for example in code documentation or in descriptions posted in online forums. Such summaries are extremely useful for applications such as code search but are expensive to manually author, hence only done for a small fraction of all code that is produced. In this paper, we present the first completely datadriven approach for generating high level summaries of source code. Our model, CODE-NN , uses Long Short Term Memory (LSTM) networks with attention to produce sentences that describe C# code snippets and SQL queries. CODE-NN is trained on a new corpus that is automatically collected from StackOverflow, which we release. Experiments demonstrate strong performance on two tasks: (1) code summarization, where we establish the first end-to-end learning results and outperform strong baselines, and (2) code retrieval, where our learned model improves the state of the art on a recently introduced C# benchmark by a large margin. <s> BIB007 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Code to Text and Text to Code <s> Relational databases store a significant amount of the worlds data. However, accessing this data currently requires users to understand a query language such as SQL. We propose Seq2SQL, a deep neural network for translating natural language questions to corresponding SQL queries. Our model uses rewards from in the loop query execution over the database to learn a policy to generate the query, which contains unordered parts that are less suitable for optimization via cross entropy loss. Moreover, Seq2SQL leverages the structure of SQL to prune the space of generated queries and significantly simplify the generation problem. In addition to the model, we release WikiSQL, a dataset of 80654 hand-annotated examples of questions and SQL queries distributed across 24241 tables fromWikipedia that is an order of magnitude larger than comparable datasets. By applying policy based reinforcement learning with a query execution environment to WikiSQL, Seq2SQL outperforms a state-of-the-art semantic parser, improving execution accuracy from 35.9% to 59.4% and logical form accuracy from 23.4% to 48.3%. <s> BIB008 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Code to Text and Text to Code <s> We consider the problem of parsing natural language descriptions into source code written in a general-purpose programming language like Python. Existing data-driven methods treat this problem as a language generation task without considering the underlying syntax of the target programming language. Informed by previous work in semantic parsing, in this paper we propose a novel neural architecture powered by a grammar model to explicitly capture the target syntax as prior knowledge. Experiments find this an effective way to scale up to generation of complex programs from natural language descriptions, achieving state-of-the-art results that well outperform previous code generation and semantic parsing approaches. <s> BIB009 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Code to Text and Text to Code <s> We present new data and semantic parsing methods for the problem of mapping English sentences to Bash commands (NL2Bash). Our long-term goal is to enable any user to perform operations such as file manipulation, search, and application-specific scripting by simply stating their goals in English. We take a first step in this domain, by providing a new dataset of challenging but commonly used Bash commands and expert-written English descriptions, along with baseline methods to establish performance levels on this task. <s> BIB010
Linking natural language text to source code has many useful applications, such as program synthesis, traceability, search, and documentation. However, the diversity of both text and code, the ambiguity of text, the compositional nature of code, and the layered abstractions in software make interconnecting text and code a hard problem. Probabilistic machine-learning models provide a principled method for modeling and resolving ambiguities in text and in code. Generating natural language from source code, i.e., code-to-text, has applications to code documentation and readability. For example, Oda et al. BIB004 translate Python code to pseudocode (in natural language) using machine translation techniques, with the goal of producing a more readable generation of the code. Iyer et al. BIB007 design a neural attention model that summarizes code as text. Movshovitz-Attias and Cohen BIB001 generate comments from code using n-gram models and topic models. The reverse direction, text-to-code, aims to help people, both developers and end users, write programs more easily. This area is closely related to semantic parsing in NLP. Semantic parsing is the task of converting a natural language utterance into a representation of its meaning, often database or logical queries that could subsequently be used for question answering  . We do not have space to fully describe the large body of work that has been done in semantic parsing in NLP but instead focus on text-to-code methods that output code in languages that are used by human software developers. This area has attracted growing interest, with applications such as converting natural language to Excel macros BIB003 , Java expressions BIB005 , shell commands  BIB010 , simple if-then programs BIB006 , regular expressions BIB002 , and to SQL queries BIB008 . Finally, Yin and Neubig BIB009 have recently presented a neural architecture for general-purpose code generation. For more details, Neubig  provides an informal survey of code generation methods.