A Survey of Machine Learning for Big Code and Naturalness <s> Types of Code Generating Models. <s> Human evaluations of machine translation are extensive but expensive. Human evaluations can take months to finish and involve human labor that can not be reused. We propose a method of automatic machine translation evaluation that is quick, inexpensive, and language-independent, that correlates highly with human evaluation, and that has little marginal cost per run. We present this method as an automated understudy to skilled human judges which substitutes for them when there is need for quick or frequent evaluations. <s> BIB001 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Types of Code Generating Models. <s> Phrase-based statistical machine translation approaches have been highly successful in translating between natural languages and are heavily used by commercial systems (e.g. Google Translate). The main objective of this work is to investigate the applicability of these approaches for translating between programming languages. Towards that, we investigated several variants of the phrase-based translation approach: i) a direct application of the approach to programming languages, ii) a novel modification of the approach to incorporate the grammatical structure of the target programming language (so to avoid generating target programs which do not parse), and iii) a combination of ii) with custom rules added to improve the quality of the translation. To experiment with the above systems, we investigated machine translation from C# to Java. For the training, which takes about 60 hours, we used a parallel corpus of 20,499 C#-to-Java method translations. We then evaluated each of the three systems above by translating 1,000 C# methods. Our experimental results indicate that with the most advanced system, about 60% of the translated methods compile (the top ranked) and out of a random sample of 50 correctly compiled methods, 68% (34 methods) were semantically equivalent to the reference solution. <s> BIB002 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Types of Code Generating Models. <s> Millions of computer end users need to perform tasks over tabular spreadsheet data, yet lack the programming knowledge to do such tasks automatically. This paper describes the design and implementation of a robust natural language based interface to spreadsheet programming. Our methodology involves designing a typed domain-specific language (DSL) that supports an expressive algebra of map, filter, reduce, join, and formatting capabilities at a level of abstraction appropriate for non-expert users. The key algorithmic component of our methodology is a translation algorithm for converting a natural language specification in the context of a given spreadsheet to a ranked set of likely programs in the DSL. The translation algorithm leverages the spreadsheet spatial and temporal context to assign interpretations to specifications with implicit references, and is thus robust to a variety of ways in which end users can express the same task. The translation algorithm builds over ideas from keyword programming and semantic parsing to achieve both high precision and high recall. We implemented the system as an Excel add-in called NLyze that supports a rich user interaction model including annotating the user's natural language specification and explaining the synthesized DSL programs by paraphrasing them into structured English. We collected a total of 3570 English descriptions for 40 spreadsheet tasks and our system was able to generate the intended interpretation as the top candidate for 94% (97 for the top 3) of those instances. <s> BIB003 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Types of Code Generating Models. <s> In this paper, we have tried to use Statistical machine translation in order to convert Python 2 code to Python 3 code. We use data from two projects and achieve a high BLEU score. We also investigate the cross-project training and testing to analyze the errors so as to ascertain differences with previous case. We have described a pilot study on modeling programming languages as natural language to build translation models on the lines of natural languages. This can be further worked on to translate between versions of a programming language or crossprogramming-languages code translation. <s> BIB004 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Types of Code Generating Models. <s> Prior research shows that directly applying phrase-based SMT on lexical tokens to migrate Java to C# produces much semantically incorrect code. A key limitation is the use of sequences in phrase-based SMT to model and translate source code with well-formed structures. We propose mppSMT, a divide-and-conquer technique to address that with novel training and migration algorithms using phrase-based SMT in three phases. First, mppSMT treats a program as a sequence of syntactic units and maps/translates such sequences in two languages to one another. Second, with a syntax-directed fashion, it deals with the tokens within syntactic units by encoding them with semantic symbols to represent their data and token types. This encoding via semantic symbols helps better migration of API usages. Third, the lexical tokens corresponding to each sememe are mapped or migrated. The resulting sequences of tokens are merged together to form the final migrated code. Such divide-and-conquer and syntax-direction strategies enable phrase-based SMT to adapt well to syntactical structures in source code, thus, improving migration accuracy. Our empirical evaluation on several real-world systems shows that 84.8 -- 97.9% and 70 -- 83% of the migrated methods are syntactically and semantically correct, respectively. 26.3 -- 51.2% of total migrated methods are exactly matched to the human-written C# code in the oracle. Compared to Java2CSharp, a rule-based migration tool, it achieves higher semantic accuracy from 6.6 -- 57.7% relatively. Importantly, it does not require manual labeling for training data or manual definition of rules. <s> BIB005 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Types of Code Generating Models. <s> Pseudo-code written in natural language can aid the comprehension of source code in unfamiliar programming languages. However, the great majority of source code has no corresponding pseudo-code, because pseudo-code is redundant and laborious to create. If pseudo-code could be generated automatically and instantly from given source code, we could allow for on-demand production of pseudo-code without human effort. In this paper, we propose a method to automatically generate pseudo-code from source code, specifically adopting the statistical machine translation (SMT) framework. SMT, which was originally designed to translate between two natural languages, allows us to automatically learn the relationship between source code/pseudo-code pairs, making it possible to create a pseudo-code generator with less human effort. In experiments, we generated English or Japanese pseudo-code from Python statements using SMT, and find that the generated pseudo-code is largely accurate, and aids code understanding. <s> BIB006 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Types of Code Generating Models. <s> We consider the problem of building probabilistic models that jointly model short natural language utterances and source code snippets. The aim is to bring together recent work on statistical modelling of source code and work on bimodal models of images and natural language. The resulting models are useful for a variety of tasks that involve natural language and source code. We demonstrate their performance on two retrieval tasks: retrieving source code snippets given a natural language query, and retrieving natural language descriptions given a source code query (i.e., source code captioning). Experiments show there to be promise in this direction, and that modelling the structure of source code improves performance. <s> BIB007 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Types of Code Generating Models. <s> We present a novel technique for automatic program correction in MOOCs, capable of fixing both syntactic and semantic errors without manual, problem specific correction strategies. Given an incorrect student program, it generates candidate programs from a distribution of likely corrections, and checks each candidate for correctness against a test suite. The key observation is that in MOOCs many programs share similar code fragments, and the seq2seq neural network model, used in the natural-language processing task of machine translation, can be modified and trained to recover these fragments. Experiment shows our scheme can correct 29% of all incorrect submissions and out-performs state of the art approach which requires manual, problem specific correction strategies. <s> BIB008 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Types of Code Generating Models. <s> We present a neural encoder-decoder model to convert images into presentational markup based on a scalable coarse-to-fine attention mechanism. Our method is evaluated in the context of image-to-LaTeX generation, and we introduce a new dataset of real-world rendered mathematical expressions paired with LaTeX markup. We show that unlike neural OCR techniques using CTC-based models, attention-based approaches can tackle this non-standard OCR task. Our approach outperforms classical mathematical OCR systems by a large margin on in-domain rendered data, and, with pretraining, also performs well on out-of-domain handwritten data. To reduce the inference complexity associated with the attention-based approaches, we introduce a new coarse-to-fine attention layer that selects a support region before applying attention. <s> BIB009 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Types of Code Generating Models. <s> Many language generation tasks require the production of text conditioned on both structured and unstructured inputs. We present a novel neural network architecture which generates an output sequence conditioned on an arbitrary number of input functions. Crucially, our approach allows both the choice of conditioning context and the granularity of generation, for example characters or tokens, to be marginalised, thus permitting scalable and effective training. Using this framework, we address the problem of generating programming code from a mixed natural language and structured specification. We create two new data sets for this paradigm derived from the collectible trading card games Magic the Gathering and Hearthstone. On these, and a third preexisting corpus, we demonstrate that marginalising multiple predictors allows our model to outperform strong benchmarks. <s> BIB010 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Types of Code Generating Models. <s> We consider the problem of parsing natural language descriptions into source code written in a general-purpose programming language like Python. Existing data-driven methods treat this problem as a language generation task without considering the underlying syntax of the target programming language. Informed by previous work in semantic parsing, in this paper we propose a novel neural architecture powered by a grammar model to explicitly capture the target syntax as prior knowledge. Experiments find this an effective way to scale up to generation of complex programs from natural language descriptions, achieving state-of-the-art results that well outperform previous code generation and semantic parsing approaches. <s> BIB011 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Types of Code Generating Models. <s> We introduce a model that learns to convert simple hand drawings into graphics programs written in a subset of \LaTeX. The model combines techniques from deep learning and program synthesis. We learn a convolutional neural network that proposes plausible drawing primitives that explain an image. These drawing primitives are like a trace of the set of primitive commands issued by a graphics program. We learn a model that uses program synthesis techniques to recover a graphics program from that trace. These programs have constructs like variable bindings, iterative loops, or simple kinds of conditionals. With a graphics program in hand, we can correct errors made by the deep network, measure similarity between drawings by use of similar high-level geometric structures, and extrapolate drawings. Taken together these results are a step towards agents that induce useful, human-readable programs from perceptual input. <s> BIB012 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Types of Code Generating Models. <s> Transforming a graphical user interface screenshot created by a designer into computer code is a typical task conducted by a developer in order to build customized software, websites, and mobile applications. In this paper, we show that deep learning methods can be leveraged to train a model end-to-end to automatically generate code from a single input image with over 77% of accuracy for three different platforms (i.e. iOS, Android and web-based technologies). <s> BIB013
We use external context C(c) to refine code generating models into three subcategories. Language Models. Language models model the language itself, without using any external context, i.e., C(c) = ∅. Although LMs learn the high-level structure and constraints of programming languages fairly easily, predicting and generating source code identifiers (e.g., variable and method names), long-range dependencies, and taking into account code semantics makes the language modeling of code a hard and interesting research area. We discuss these and other differences and their implications for probabilistic modeling in Section 6. Code LMs are evaluated like LMs in NLP, using perplexity (or equivalently cross-entropy) and word error rate. Cross-entropy H is the most common measure. Language models-as most predictive machine-learning models-can be seen as compression algorithms where the model predicts the full output (i.e., decompresses) using extra information. Cross-entropy measures the average number of extra bits of information per token of code that a model needs to decompress the correct output using a perfect code (in the information-theoretic sense), where M is the number of tokens within c. By convention, the average is reported per token, even for non-token models. Thus, a "perfect" model, correctly predicting all tokens with probability 1, would require no additional bits of information because, in a sense, it already "knows" everything. Cross-entropy allows comparisons across different models. Other, application-specific measures, are used when the LM was trained for a specific task, such as code completion (Section 5.1). Code Transducer Models. Inspired by statistical machine translation (SMT), transducer models translate/transduce code from one format into another (i.e., C(c) is also code), such as translating code from one source language into another, target language. They have the form P D (c|s), where c is the target source code that is generated and C(c) = s is the source source code. Most code transducer models use phrase-based machine translation. Intuitively, phrase-based models assume that small chunks from the source input can directly be mapped to chunks in the output. Although this assumption is reasonable in NLP and many source code tasks, these models present challenges in capturing long-range dependencies within the source and target. For example, as we will mention in the next section, transducing code from an imperative source language to a functional target is not currently possible, because the source and target are related with a significantly more complicated relation that simply matching "chunks" of the input code to "chunks" in the output. These types of models have found application within code migration BIB004 BIB002 BIB005 , pseudocode generation BIB006 , and code fixing BIB008 . Traditionally transducer models have followed a noisy channel model, in which they combine a language model P D (c) of the target language with a translation/transduction model P D (s|c) to match elements between the source and the target. These methods pick the optimal transduction c * such that c * = arg max P D (c|s) = arg max P D (s|c)P D (c), where the second equality derives from the Bayes equation. Again, these probabilistic generative models of code do not necessarily produce valid code, due to the simplifying assumptions they make in both P D (c) and P D (s|c). More recently, machine translation methods based on phrase-based models and the noisy channel model have been outperformed by neural network-based methods that directly model P D (c|s). Transducer models can be evaluated with SMT evaluation measures, such as BLEU BIB001 commonly used in machine translation as an approximate measure of translation quality-or programming and logic-related measures (e.g., "Does the translated code parse/compile?" and "Are the two snippets equivalent?"). Multimodal Models. Code-generating multimodal models correlate code with one or more noncode modalities, such as comments, specifications, or search queries. These models have the form P D (c|m) i.e., C(c) = m is a representation of one or more non-code modalities. Multimodal models are closely related to representational models (discussed in Section 4. Multimodal models of code have been used for code synthesis, where the non-code modalities are leveraged to estimated a conditional generative model of code, e.g., synthesis of code given a natural language description by Gulwani and Marron BIB003 and more recently by Yin and Neubig BIB011 . The latter model is a syntactic model that accepts natural language. Recently, Beltramelli BIB013 , Deng et al. BIB009 , and Ellis et al. BIB012 designed multimodal model that accept visual input (the non-code modality) and generate code in a DSL describing how the input (hand-drawn image, GUI screenshot) was constructed. Another use of these models is to score the co-appearance of the modalities, e.g., in code search, to score the probability of some text given a textual query BIB007 . This stream of research is related to work in NLP and computer vision where one seeks to generate a natural language description for an image. These models are closely related to the other code generating models, since they generate code. These models make also assume that the input modality conditions the generation process. Multimodal models combine an assumption with a design choice. Like language models, these models assume that probabilistic models can capture the process by which developers generate code; unlike language models, they additionally bias code generation using information from the input modality m. The design choice is how to transform the input modality into an intermediate representation. For example, Allamanis et al. BIB007 use a bag-of-words assumption averaging the words' distributed representations. However, this limits the expressivity of the models, because the input modality has to fit in whole within the distributed representation. To address this issue, Ling et al. BIB010 and Yin and Neubig BIB011 use neural attention mechanisms to selectively attend to information within the input modality without the need to "squash" all the information into a single representation. Finally, the text-to-code problem, in which we take the input modality m to be natural language text and the other modality c to be code, is closely related to the problem of semantic parsing in NLP; see Section 5.5.