A Survey of Machine Learning for Big Code and Naturalness <s> Pattern-mining Models of Source Code <s> Data Mining: Practical Machine Learning Tools and Techniques offers a thorough grounding in machine learning concepts as well as practical advice on applying machine learning tools and techniques in real-world data mining situations. This highly anticipated third edition of the most acclaimed work on data mining and machine learning will teach you everything you need to know about preparing inputs, interpreting outputs, evaluating results, and the algorithmic methods at the heart of successful data mining. Thorough updates reflect the technical changes and modernizations that have taken place in the field since the last edition, including new material on Data Transformations, Ensemble Learning, Massive Data Sets, Multi-instance Learning, plus a new version of the popular Weka machine learning software developed by the authors. Witten, Frank, and Hall include both tried-and-true techniques of today as well as methods at the leading edge of contemporary research. *Provides a thorough grounding in machine learning concepts as well as practical advice on applying the tools and techniques to your data mining projects *Offers concrete tips and techniques for performance improvement that work by transforming the input or output in machine learning methods *Includes downloadable Weka software toolkit, a collection of machine learning algorithms for data mining tasks-in an updated, interactive interface. Algorithms in toolkit cover: data pre-processing, classification, regression, clustering, association rules, visualization <s> BIB001 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Pattern-mining Models of Source Code <s> We present the first method for automatically mining code idioms from a corpus of previously written, idiomatic software projects. We take the view that a code idiom is a syntactic fragment that recurs across projects and has a single semantic purpose. Idioms may have metavariables, such as the body of a for loop. Modern IDEs commonly provide facilities for manually defining idioms and inserting them on demand, but this does not help programmers to write idiomatic code in languages or using libraries with which they are unfamiliar. We present Haggis, a system for mining code idioms that builds on recent advanced techniques from statistical natural language processing, namely, nonparametric Bayesian probabilistic tree substitution grammars. We apply Haggis to several of the most popular open source projects from GitHub. We present a wide range of evidence that the resulting idioms are semantically meaningful, demonstrating that they do indeed recur across software projects and that they occur more frequently in illustrative code examples collected from a Q&A site. Manual examination of the most common idioms indicate that they describe important program concepts, including object creation, exception handling, and resource management. <s> BIB002 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Pattern-mining Models of Source Code <s> Developers spend much of their time reading and browsing source code, raising new opportunities for summarization methods. Indeed, modern code editors provide code folding , which allows one to selectively hide blocks of code. However this is impractical to use as folding decisions must be made manually or based on simple rules. We introduce the autofolding problem , which is to automatically create a code summary by folding less informative code regions. We present a novel solution by formulating the problem as a sequence of AST folding decisions, leveraging a scoped topic model for code tokens. On an annotated set of popular open source projects, we show that our summarizer outperforms simpler baselines, yielding a 28 percent error reduction. Furthermore, we find through a case study that our summarizer is strongly preferred by experienced developers. More broadly, we hope this work will aid program comprehension by turning code folding into a usable and valuable tool. <s> BIB003 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Pattern-mining Models of Source Code <s> Existing API mining algorithms are not yet practical to use as they require expensive parameter tuning and the returned set of API calls can be large, highly redundant and difficult to understand. In an attempt to remedy these shortcomings we present PAM (Probabilistic API Miner), a near parameter-free probabilistic algorithm for mining the most informative API call patterns. We show that PAM significantly outperforms both MAPO and UPMiner, achieving 70% test-set precision, at retrieving relevant API call sequences from GitHub. Moreover, we focus on libraries for which the developers have explicitly provided code examples, yielding over 300,000 LOC of hand-written API example code from the 967 client projects in the data set. This evaluation suggests that the hand-written examples actually have limited coverage of real API usages. <s> BIB004 </s> A Survey of Machine Learning for Big Code and Naturalness <s> Pattern-mining Models of Source Code <s> Many existing knowledge bases (KBs), including Freebase, Yago, and NELL, rely on a fixed ontology, given as an input to the system, which defines the data to be cataloged in the KB, i.e., a hierarchy of categories and relations between them. The system then extracts facts that match the predefined ontology. We propose an unsupervised model that jointly learns a latent ontological structure of an input corpus, and identifies facts from the corpus that match the learned structure. Our approach combines mixed membership stochastic block models and topic models to infer a structure by jointly modeling text, a latent concept hierarchy, and latent semantic relationships among the entities mentioned in the text. As a case study, we apply the model to a corpus of Web documents from the software domain, and evaluate the accuracy of the various components of the learned ontology. <s> BIB005
Pattern-mining models aim to discover a finite set of human-interpretable patterns from source code, without annotation or supervision, and present the mined patterns to software engineers. Broadly, these models cluster source code into a finite set of groups. Probabilistic pattern-mining models of code infer the likely latent structure of a probability distribution where д is a deterministic function that returns a (possibly partial, e.g., API calls only) view of the code and l represents a set of latent variables that the model introduces and aims to infer. Applications of such models are common in the mining software repositories community and include documentation (e.g., API patterns), summarization, and anomaly detection. Table 3 lists this work. Unsupervised learning is one of the most challenging areas in machine learning. This hardness stems from the need to automatically distinguish important patterns in the code from spurious patterns that may appear to be significant because of limited and noisy data. When designing unsupervised models, the core assumption lies in the objective function being used, and often we resort to using a principle from statistics, information theory, or a proxy supervised task. Like These models have the general form P D (д(c)). References annotated with * are also included in other categories. all machine-learning models, they require assumptions about how the data are represented. An important issue with unsupervised methods is the hardness of evaluating the output, since the quality of the output is rarely quantifiable. A vast literature on non-probabilistic methods exploits data-mining methods, such as frequent pattern mining and anomaly detection BIB001 . We do not discuss these models here, since they are not probabilistic models of code. Classic probabilistic topic models  , which usually views code (or other software engineering artifacts) as a bag-ofwords, have also been heavily investigated. Since these models and their strengths and limitations are well understood, we omit them here. Allamanis and Sutton BIB002 learn a tree substitution grammar (TSG) using Bayesian nonparametrics, a technique originally devised for natural language grammars. TSGs learn to group commonly co-appearing grammar productions (tree fragments). Although TSGs have been used in NLP to improve parsing performance (which is ambiguous in text), Allamanis and Sutton BIB002 observe that the inferred fragments represent common code usage conventions and name them idioms. Later, Allamanis et al.  extend this technique to mine semantic code idioms by modifying the input code representation and adapting the inference method. In a similar fashion, Fowkes and Sutton BIB004 learn the latent variables of a graphical model to infer common API usage patterns. Their method automatically infers the most probable grouping of API elements. This is in stark contrast to frequency-based methods  that suffer from finding frequent but not necessarily "interesting" patterns. Finally, Movshovitz-Attias and Cohen BIB005 infer the latent variables of a graphical model that models a software ontology. As in NLP and machine learning in general, evaluating pattern-mining models is hard, since the quality of the discovered latent structure is subjective. Thus, researchers often resort to extrinsic, application-specific measures. For example, Fowkes et al. BIB003 run a user study to directly assess the quality of their summarization method.