A Survey of Machine Learning for Big Code and Naturalness <s> Code Translation, Copying, and Clones <s> Human evaluations of machine translation are extensive but expensive. Human evaluations can take months to finish and involve human labor that can not be reused. We propose a method of automatic machine translation evaluation that is quick, inexpensive, and language-independent, that correlates highly with human evaluation, and that has little marginal cost per run. We present this method as an automated understudy to skilled human judges which substitutes for them when there is need for quick or frequent evaluations. <s> BIB001 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Code Translation, Copying, and Clones <s> Summary. This report summarizes my overview talk on software clone detection research. It first discusses the notion of software redundancy, cloning, duplication, and similarity. Then, it describes various categorizations of clone types, empirical studies on the root causes for cloning, current opinions and wisdom of consequences of cloning, empirical studies on the evolution of clones, ways to remove, to avoid, and to detect them, empirical evaluations of existing automatic clone detector performance (such as recall, precision, time and space consumption) and their fitness for a particular purpose, benchmarks for clone detector evaluations, presentation issues, and last but not least application of clone detection in other related fields. After each summary of a subarea, I am listing open research questions. <s> BIB002 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Code Translation, Copying, and Clones <s> Phrase-based statistical machine translation approaches have been highly successful in translating between natural languages and are heavily used by commercial systems (e.g. Google Translate). The main objective of this work is to investigate the applicability of these approaches for translating between programming languages. Towards that, we investigated several variants of the phrase-based translation approach: i) a direct application of the approach to programming languages, ii) a novel modification of the approach to incorporate the grammatical structure of the target programming language (so to avoid generating target programs which do not parse), and iii) a combination of ii) with custom rules added to improve the quality of the translation. To experiment with the above systems, we investigated machine translation from C# to Java. For the training, which takes about 60 hours, we used a parallel corpus of 20,499 C#-to-Java method translations. We then evaluated each of the three systems above by translating 1,000 C# methods. Our experimental results indicate that with the most advanced system, about 60% of the translated methods compile (the top ranked) and out of a random sample of 50 correctly compiled methods, 68% (34 methods) were semantically equivalent to the reference solution. <s> BIB003 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Code Translation, Copying, and Clones <s> Prior research shows that directly applying phrase-based SMT on lexical tokens to migrate Java to C# produces much semantically incorrect code. A key limitation is the use of sequences in phrase-based SMT to model and translate source code with well-formed structures. We propose mppSMT, a divide-and-conquer technique to address that with novel training and migration algorithms using phrase-based SMT in three phases. First, mppSMT treats a program as a sequence of syntactic units and maps/translates such sequences in two languages to one another. Second, with a syntax-directed fashion, it deals with the tokens within syntactic units by encoding them with semantic symbols to represent their data and token types. This encoding via semantic symbols helps better migration of API usages. Third, the lexical tokens corresponding to each sememe are mapped or migrated. The resulting sequences of tokens are merged together to form the final migrated code. Such divide-and-conquer and syntax-direction strategies enable phrase-based SMT to adapt well to syntactical structures in source code, thus, improving migration accuracy. Our empirical evaluation on several real-world systems shows that 84.8 -- 97.9% and 70 -- 83% of the migrated methods are syntactically and semantically correct, respectively. 26.3 -- 51.2% of total migrated methods are exactly matched to the human-written C# code in the oracle. Compared to Java2CSharp, a rule-based migration tool, it achieves higher semantic accuracy from 6.6 -- 57.7% relatively. Importantly, it does not require manual labeling for training data or manual definition of rules. <s> BIB004 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Code Translation, Copying, and Clones <s> Code clone detection is an important problem for software maintenance and evolution. Many approaches consider either structure or identifiers, but none of the existing detection techniques model both sources of information. These techniques also depend on generic, handcrafted features to represent code fragments. We introduce learning-based detection techniques where everything for representing terms and fragments in source code is mined from the repository. Our code analysis supports a framework, which relies on deep learning, for automatically linking patterns mined at the lexical level with patterns mined at the syntactic level. We evaluated our novel learning-based approach for code clone detection with respect to feasibility from the point of view of software maintainers. We sampled and manually evaluated 398 file- and 480 method-level pairs across eight real-world Java systems; 93% of the file- and method-level samples were evaluated to be true positives. Among the true positives, we found pairs mapping to all four clone types. We compared our approach to a traditional structure-oriented technique and found that our learning-based approach detected clones that were either undetected or suboptimally reported by the prominent tool Deckard. Our results affirm that our learning-based approach is suitable for clone detection and a tenable technique for researchers. <s> BIB005 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Code Translation, Copying, and Clones <s> Deep Neural Networks have been shown to succeed at a range of natural language tasks such as machine translation and text summarization. While tasks on source code (ie, formal languages) have been considered recently, most work in this area does not attempt to capitalize on the unique opportunities offered by its known syntax and structure. In this work, we introduce SmartPaste, a first task that requires to use such information. The task is a variant of the program repair problem that requires to adapt a given (pasted) snippet of code to surrounding, existing source code. As first solutions, we design a set of deep neural models that learn to represent the context of each variable location and variable usage in a data flow-sensitive way. Our evaluation suggests that our models can learn to solve the SmartPaste task in many cases, achieving 58.6% accuracy, while learning meaningful representation of variable usages. <s> BIB006
The success of statistical machine translation (SMT) among (natural) languages has inspired researchers to use machine learning to translate code from one source language (e.g., Java) to another (e.g., C#). Although rule-based rewriting systems can be (and have been) used, it is tedious to create and maintain these rules in the face of language evolution. SMT models are well suited for this task, although they tend to produce invalid code. To reduce these errors during translation, Karaivanov et al. BIB003 and Nguyen et al. BIB004 add semantic constraints to the translation process. Existing research has applied widely used SMT models for text. Although these models learn mappings between different language constructs such as APIs, they have only been used for translating between programming languages of similar paradigms and structure (C# and Java are both object-oriented languages with managed memory). This is an important limitation; machinelearning innovations are required to translate between languages of different types (e.g., Java to Haskell or assembly to C) or even languages with different memory management (e.g., Java to C). Existing per-statement SMT from Java to C does not track memory allocations and therefore fails to emit memory de-allocations that C's lack of garbage collection requires. Similarly, translating object-oriented code to functional languages will require learning the conceptual differences of the two paradigms while preserving semantics, such as learning to translate a loop to a map-reduce functional. Researchers evaluate translation models by scoring exact matches, measuring the syntactic or semantic correctness of the translated code or using BLEU BIB001 . We discuss this and other measure-related issues in Section 6. Developers often copy code during development. This practice requires fixups to rename variables and handle name collisions; it can also create code clones, similar code snippets in different locations of a code base BIB002 . Allamanis and Brockschmidt BIB006 automate naming cleanups after copying; they use structured prediction and distributed representations to adapt/port a pasted snippet's variables into the target context. Their method probabilistically represents semantic information about variable use to predict the correct name adaptations without external information (e.g., tests). Clones may indicate refactoring opportunities (that allow reusing the cloned code). White et al. BIB005 use autoencoders and recurrent neural networks  to find clones as code snippets that share similar distributed representations. Using distributed vector representations allows them to learn a continuous similarity metric between code locations rather than using edit distance.