A Survey of Machine Learning for Big Code and Naturalness <s> Code-generating Probabilistic Models of Source Code <s> We study the problem of building generative models of natural source code (NSC); that is, source code written and understood by humans. Our primary contribution is to describe a family of generative models for NSC that have three key properties: First, they incorporate both sequential and hierarchical structure. Second, we learn a distributed representation of source code elements. Finally, they integrate closely with a compiler, which allows leveraging compiler logic and abstractions when building structure into the model. We also develop an extension that includes more complex structure, refining how the model generates identifier tokens based on what variables are currently in scope. Our models can be learned efficiently, and we show empirically that including appropriate structure greatly improves the models, measured by the probability of generating test programs. <s> BIB001 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Code-generating Probabilistic Models of Source Code <s> Natural languages like English are rich, complex, and powerful. The highly creative and graceful use of languages like English and Tamil, by masters like Shakespeare and Avvaiyar, can certainly delight and inspire. But in practice, given cognitive constraints and the exigencies of daily life, most human utterances are far simpler and much more repetitive and predictable. In fact, these utterances can be very usefully modeled using modern statistical methods. This fact has led to the phenomenal success of statistical approaches to speech recognition, natural language translation, question-answering, and text mining and comprehension. We begin with the conjecture that most software is also natural, in the sense that it is created by humans at work, with all the attendant constraints and limitations---and thus, like natural language, it is also likely to be repetitive and predictable. We then proceed to ask whether (a) code can be usefully modeled by statistical language models and (b) such models can be leveraged to support software engineers. Using the widely adopted n-gram model, we provide empirical evidence supportive of a positive answer to both these questions. We show that code is also very regular, and, in fact, even more so than natural languages. As an example use of the model, we have developed a simple code completion engine for Java that, despite its simplicity, already improves Eclipse's completion capability. We conclude the paper by laying out a vision for future research in this area. <s> BIB002
Code-generating probabilistic models of code are probability distributions that describe a stochastic process for generating valid code, i.e., they model how code is written. Given training data D, an output code representation c, and a possibly empty context C(c), these models learn the probability distribution P D (c|C(c)) and sample P D to generate code. When C(c) = ∅, the probability distribution P D is a language model of code, i.e., it models how code is generated when no external context information is available. When C(c) is a non-code modality (e.g., natural language), P D describes a code-generative multimodal model of code. When C(c) is also code, the probability distribution P D is a transducer model of code. In addition to generating code, by definition, code generating probabilistic models act as a scoring function, assigning a non-zero probability to every possible snippet of code. This score, sometimes referred to as "naturalness" of the code BIB002 , suggests how probable the code is under a learned model. Since code-generating models predict the complex structure of code, they make simplifying assumptions about the generative process and iteratively predict elements of the code to generate a full code unit, e.g., code file or method. Because of code's structural complexity and the simplifying assumptions these models make to cope with it, none of the existing models in the literature generate code that always parses, compiles, and executes. Some of the models do, however, impose constraints that take code structure into account to remove some inconsistencies; for instance, BIB001 only generate variables declared within each scope. We structure the discussion about code-generating models of code as follows. We first discuss how models in this category generate code, and then we show how the three different types of models (language, transducer, and multimodal models) differ. (Table 1) lists related work on codegenerating models.