A Survey of Machine Learning for Big Code and Naturalness <s> The Third Wave of Machine Learning <s> In this paper an attempt is made to explore the logical foundations of computer programming by use of techniques which were first applied in the study of geometry and have later been extended to other branches of mathematics. This involves the elucidation of sets of axioms and rules of inference which can be used in proofs of the properties of computer programs. Examples are given of such axioms and rules, and a formal proof of a simple theorem is displayed. Finally, it is argued that important advantages, both theoretical and practical, may follow from a pursuance of these topics. <s> BIB001 </s> A Survey of Machine Learning for Big Code and Naturalness <s> The Third Wave of Machine Learning <s> Proving that particular methods within a code base are functionally pure--deterministic and side-effect free--would aid verification of security properties including function invertibility, reproducibility of computation, and safety of untrusted code execution. Until now it has not been possible to automatically prove a method is functionally pure within a high-level imperative language in wide use, such as Java. We discuss a technique to prove that methods are functionally pure by writing programs in a subset of Java called Joe-E; a static verifier ensures that programs fall within the subset. In Joe-E, pure methods can be trivially recognized from their method signature. To demonstrate the practicality of our approach, we refactor an AES library, an experimental voting machine implementation, and an HTML parser to use our techniques. We prove that their top-level methods are verifiably pure and show how this provides high-level security guarantees about these routines. Our approach to verifiable purity is an attractive way to permit functional-style reasoning about security properties while leveraging the familiarity, convenience, and legacy code of imperative languages. <s> BIB002 </s> A Survey of Machine Learning for Big Code and Naturalness <s> The Third Wave of Machine Learning <s> At Brown University, there is excitement of having access to the Brown Corpus, containing one million English words. Since then, we have seen several notable corpora that are about 100 times larger, and in 2006, Google released a trillion-word corpus with frequency counts for all sequences up to five words long. In some ways this corpus is a step backwards from the Brown Corpus: it's taken from unfiltered Web pages and thus contains incomplete sentences, spelling errors, grammatical errors, and all sorts of other errors. It's not annotated with carefully hand-corrected part-of-speech tags. But the fact that it's a million times larger than the Brown Corpus outweighs these drawbacks. A trillion-word corpus - along with other Web-derived corpora of millions, billions, or trillions of links, videos, images, tables, and user interactions - captures even very rare aspects of human behavior. So, this corpus could serve as the basis of a complete model for certain tasks - if only we knew how to extract the model from the data. <s> BIB003 </s> A Survey of Machine Learning for Big Code and Naturalness <s> The Third Wave of Machine Learning <s> In recent years, a great many methods of learning from multi-view data by considering the diversity of different views have been proposed. These views may be obtained from multiple sources or different feature subsets. In trying to organize and highlight similarities and differences between the variety of multi-view learning approaches, we review a number of representative multi-view learning algorithms in different areas and classify them into three groups: 1) co-training, 2) multiple kernel learning, and 3) subspace learning. Notably, co-training style algorithms train alternately to maximize the mutual agreement on two distinct views of the data; multiple kernel learning algorithms exploit kernels that naturally correspond to different views and combine kernels either linearly or non-linearly to improve learning performance; and subspace learning algorithms aim to obtain a latent subspace shared by multiple views by assuming that the input views are generated from this latent subspace. Though there is significant variance in the approaches to integrating multiple views to improve learning performance, they mainly exploit either the consensus principle or the complementary principle to ensure the success of multi-view learning. Since accessing multiple views is the fundament of multi-view learning, with the exception of study on learning a model from multiple views, it is also valuable to study how to construct multiple views and how to evaluate these views. Overall, by exploring the consistency and complementary properties of different views, multi-view learning is rendered more effective, more promising, and has better generalization ability than single-view learning. <s> BIB004 </s> A Survey of Machine Learning for Big Code and Naturalness <s> The Third Wave of Machine Learning <s> We study the problem of building generative models of natural source code (NSC); that is, source code written and understood by humans. Our primary contribution is to describe a family of generative models for NSC that have three key properties: First, they incorporate both sequential and hierarchical structure. Second, we learn a distributed representation of source code elements. Finally, they integrate closely with a compiler, which allows leveraging compiler logic and abstractions when building structure into the model. We also develop an extension that includes more complex structure, refining how the model generates identifier tokens based on what variables are currently in scope. Our models can be learned efficiently, and we show empirically that including appropriate structure greatly improves the models, measured by the probability of generating test programs. <s> BIB005 </s> A Survey of Machine Learning for Big Code and Naturalness <s> The Third Wave of Machine Learning <s> We present a new approach for predicting program properties from massive codebases (aka "Big Code"). Our approach first learns a probabilistic model from existing data and then uses this model to predict properties of new, unseen programs. The key idea of our work is to transform the input program into a representation which allows us to phrase the problem of inferring program properties as structured prediction in machine learning. This formulation enables us to leverage powerful probabilistic graphical models such as conditional random fields (CRFs) in order to perform joint prediction of program properties. As an example of our approach, we built a scalable prediction engine called JSNice for solving two kinds of problems in the context of JavaScript: predicting (syntactic) names of identifiers and predicting (semantic) type annotations of variables. Experimentally, JSNice predicts correct names for 63% of name identifiers and its type annotation predictions are correct in 81% of the cases. In the first week since its release, JSNice was used by more than 30,000 developers and in only few months has become a popular tool in the JavaScript developer community. By formulating the problem of inferring program properties as structured prediction and showing how to perform both learning and inference in this context, our work opens up new possibilities for attacking a wide range of difficult problems in the context of "Big Code" including invariant generation, decompilation, synthesis and others. <s> BIB006 </s> A Survey of Machine Learning for Big Code and Naturalness <s> The Third Wave of Machine Learning <s> Program analysis tools often produce undesirable output due to various approximations. We present an approach and a system EUGENE that allows user feedback to guide such approximations towards producing the desired output. We formulate the problem of user-guided program analysis in terms of solving a combination of hard rules and soft rules: hard rules capture soundness while soft rules capture degrees of approximations and preferences of users. Our technique solves the rules using an off-the-shelf solver in a manner that is sound (satisfies all hard rules), optimal (maximally satisfies soft rules), and scales to real-world analyses and programs. We evaluate EUGENE on two different analyses with labeled output on a suite of seven Java programs of size 131–198 KLOC. We also report upon a user study involving nine users who employ EUGENE to guide an information-flow analysis on three Java micro-benchmarks. In our experiments, EUGENE significantly reduces misclassified reports upon providing limited amounts of feedback. <s> BIB007 </s> A Survey of Machine Learning for Big Code and Naturalness <s> The Third Wave of Machine Learning <s> The question of how procedural knowledge is represented and inferred is a fundamental problem in machine learning and artificial intelligence. Recent work on program induction has proposed neural architectures, based on abstractions like stacks, Turing machines, and interpreters, that operate on abstract computational machines or on execution traces. But the recursive abstraction that is central to procedural knowledge is perhaps most naturally represented by symbolic representations that have syntactic structure, such as logical expressions and source code. Combining abstract, symbolic reasoning with continuous neural reasoning is a grand challenge of representation learning. As a step in this direction, we propose a new architecture, called neural equivalence networks, for the problem of learning continuous semantic representations of mathematical and logical expressions. These networks are trained to represent semantic equivalence, even of expressions that are syntactically very different. The challenge is that semantic representations must be computed in a syntax-directed manner, because semantics is compositional, but at the same time, small changes in syntax can lead to very large changes in semantics, which can be difficult for continuous neural architectures. We perform an exhaustive evaluation on the task of checking equivalence on a highly diverse class of symbolic algebraic and boolean expression types, showing that our model significantly outperforms existing architectures. <s> BIB008 </s> A Survey of Machine Learning for Big Code and Naturalness <s> The Third Wave of Machine Learning <s> We study the effectiveness of neural sequence models for premise selection in automated theorem proving, one of the main bottlenecks in the formalization of mathematics. We propose a two stage approach for this task that yields good results for the premise selection task on the Mizar corpus while avoiding the hand-engineered features of existing state-of-the-art models. To our knowledge, this is the first time deep learning has been applied to theorem proving on a large scale. <s> BIB009 </s> A Survey of Machine Learning for Big Code and Naturalness <s> The Third Wave of Machine Learning <s> We develop a first line of attack for solving programming competition-style problems from input-output examples using deep learning. The approach is to train a neural network to predict properties of the program that generated the outputs from the inputs. We use the neural network's predictions to augment search techniques from the programming languages community, including enumerative search and an SMT-based solver. Empirically, we show that our approach leads to an order of magnitude speedup over the strong non-augmented baselines and a Recurrent Neural Network approach, and that we are able to solve problems of difficulty comparable to the simplest problems on programming competition websites. <s> BIB010 </s> A Survey of Machine Learning for Big Code and Naturalness <s> The Third Wave of Machine Learning <s> We describe a question answering model that applies to both images and structured knowledge bases. The model uses natural language strings to automatically assemble neural networks from a collection of composable modules. Parameters for these modules are learned jointly with network-assembly parameters via reinforcement learning, with only (world, question, answer) triples as supervision. Our approach, which we term a dynamic neural model network, achieves state-of-the-art results on benchmark datasets in both visual and structured domains. <s> BIB011 </s> A Survey of Machine Learning for Big Code and Naturalness <s> The Third Wave of Machine Learning <s> Distributional models that learn rich semantic word representations are a success story of recent NLP research. However, developing models that learn useful representations of phrases and sentences has proved far harder. We propose using the definitions found in everyday dictionaries as a means of bridging this gap between lexical and phrasal semantics. Neural language embedding models can be effectively trained to map dictionary definitions (phrases) to (lexical) representations of the words defined by those definitions. We present two applications of these architectures: 'reverse dictionaries' that return the name of a concept given a definition or description and general-knowledge crossword question answerers. On both tasks, neural language embedding models trained on definitions from a handful of freely-available lexical resources perform as well or better than existing commercial systems that rely on significant task-specific engineering. The results highlight the effectiveness of both neural embedding architectures and definition-based training for developing models that understand phrases and sentences. <s> BIB012 </s> A Survey of Machine Learning for Big Code and Naturalness <s> The Third Wave of Machine Learning <s> We introduce a new task, visual sense disambiguation for verbs: given an image and a verb, assign the correct sense of the verb, i.e., the one that describes the action depicted in the image. Just as textual word sense disambiguation is useful for a wide range of NLP tasks, visual sense disambiguation can be useful for multimodal tasks such as image retrieval, image description, and text illustration. We introduce VerSe, a new dataset that augments existing multimodal datasets (COCO and TUHOI) with sense labels. We propose an unsupervised algorithm based on Lesk which performs visual sense disambiguation using textual, visual, or multimodal embeddings. We find that textual embeddings perform well when gold-standard textual annotations (object labels and image descriptions) are available, while multimodal embeddings perform well on unannotated images. We also verify our findings by using the textual and multimodal embeddings as features in a supervised setting and analyse the performance of visual sense disambiguation task. VerSe is made publicly available and can be downloaded at: this https URL <s> BIB013 </s> A Survey of Machine Learning for Big Code and Naturalness <s> The Third Wave of Machine Learning <s> While researchers develop many new exciting code recommender systems, such as method-call completion, code-snippet completion, or code search, an accurate evaluation of such systems is always a challenge. We analyzed the current literature and found that most of the current evaluations rely on artificial queries extracted from released code, which begs the question: Do such evaluations reflect real-life usages? To answer this question, we capture 6,189 fine-grained development histories from real IDE interactions. We use them as a ground truth and extract 7,157 real queries for a specific method-call recommender system. We compare the results of such real queries with different artificial evaluation strategies and check several assumptions that are repeatedly used in research, but never empirically evaluated. We find that an evolving context that is often observed in practice has a major effect on the prediction quality of recommender systems, but is not commonly reflected in artificial evaluations. <s> BIB014 </s> A Survey of Machine Learning for Big Code and Naturalness <s> The Third Wave of Machine Learning <s> While deep learning is an exciting new technique, the benefits of this method need to be assessed with respect to its computational cost. This is particularly important for deep learning since these learners need hours (to weeks) to train the model. Such long training time limits the ability of (a)~a researcher to test the stability of their conclusion via repeated runs with different random seeds; and (b)~other researchers to repeat, improve, or even refute that original work. For example, recently, deep learning was used to find which questions in the Stack Overflow programmer discussion forum can be linked together. That deep learning system took 14 hours to execute. We show here that applying a very simple optimizer called DE to fine tune SVM, it can achieve similar (and sometimes better) results. The DE approach terminated in 10 minutes; i.e. 84 times faster hours than deep learning method. We offer these results as a cautionary tale to the software analytics community and suggest that not every new innovation should be applied without critical analysis. If researchers deploy some new and expensive process, that work should be baselined against some simpler and faster alternatives. <s> BIB015 </s> A Survey of Machine Learning for Big Code and Naturalness <s> The Third Wave of Machine Learning <s> Current statistical language modeling techniques, including deep-learning based models, have proven to be quite effective for source code. We argue here that the special properties of source code can be exploited for further improvements. In this work, we enhance established language modeling approaches to handle the special challenges of modeling source code, such as: frequent changes, larger, changing vocabularies, deeply nested scopes, etc. We present a fast, nested language modeling toolkit specifically designed for software, with the ability to add & remove text, and mix & swap out many models. Specifically, we improve upon prior cache-modeling work and present a model with a much more expansive, multi-level notion of locality that we show to be well-suited for modeling software. We present results on varying corpora in comparison with traditional N-gram, as well as RNN, and LSTM deep-learning language models, and release all our source code for public use. Our evaluations suggest that carefully adapting N-gram models for source code can yield performance that surpasses even RNN and LSTM based deep-learning models. <s> BIB016 </s> A Survey of Machine Learning for Big Code and Naturalness <s> The Third Wave of Machine Learning <s> We introduce neural networks for end-to-end differentiable proving of queries to knowledge bases by operating on dense vector representations of symbols. These neural networks are constructed recursively by taking inspiration from the backward chaining algorithm as used in Prolog. Specifically, we replace symbolic unification with a differentiable computation on vector representations of symbols using a radial basis function kernel, thereby combining symbolic reasoning with learning subsymbolic vector representations. By using gradient descent, the resulting neural network can be trained to infer facts from a given incomplete knowledge base. It learns to (i) place representations of similar symbols in close proximity in a vector space, (ii) make use of such similarities to prove queries, (iii) induce logical rules, and (iv) use provided and induced logical rules for multi-hop reasoning. We demonstrate that this architecture outperforms ComplEx, a state-of-the-art neural link prediction model, on three out of four benchmark knowledge bases while at the same time inducing interpretable function-free first-order logic rules. <s> BIB017 </s> A Survey of Machine Learning for Big Code and Naturalness <s> The Third Wave of Machine Learning <s> Deep learning techniques lie at the heart of several significant AI advances in recent years including object recognition and detection, image captioning, machine translation, speech recognition and synthesis, and playing the game of Go. Automated first-order theorem provers can aid in the formalization and verification of mathematical theorems and play a crucial role in program analysis, theory reasoning, security, interpolation, and system verification. Here we suggest deep learning based guidance in the proof search of the theorem prover E. We train and compare several deep neural network models on the traces of existing ATP proofs of Mizar statements and use them to select processed clauses during proof search. We give experimental evidence that with a hybrid, two-phase approach, deep learning based guidance can significantly reduce the average number of proof search steps while increasing the number of theorems proved. Using a few proof guidance strategies that leverage deep neural networks, we have found first-order proofs of 7.36% of the first-order logic translations of the Mizar Mathematical Library theorems that did not previously have ATP generated proofs. This increases the ratio of statements in the corpus with ATP generated proofs from 56% to 59%. <s> BIB018 </s> A Survey of Machine Learning for Big Code and Naturalness <s> The Third Wave of Machine Learning <s> Learning tasks on source code (i.e., formal languages) have been considered recently, but most work has tried to transfer natural language methods and does not capitalize on the unique opportunities offered by code's known syntax. For example, long-range dependencies induced by using the same variable or function in distant locations are often not considered. We propose to use graphs to represent both the syntactic and semantic structure of code and use graph-based deep learning methods to learn to reason over program structures. ::: In this work, we present how to construct graphs from source code and how to scale Gated Graph Neural Networks training to such large graphs. We evaluate our method on two tasks: VarNaming, in which a network attempts to predict the name of a variable given its usage, and VarMisuse, in which the network learns to reason about selecting the correct variable that should be used at a given program location. Our comparison to methods that use less structured program representations shows the advantages of modeling known structure, and suggests that our models learn to infer meaningful names and to solve the VarMisuse task in many cases. Additionally, our testing showed that VarMisuse identifies a number of bugs in mature open-source projects. <s> BIB019
The first wave of machine learning for source code applied off-the-shelf machine-learning tools with hand-extracted features. The second wave, reviewed here, avoids manual feature extraction and uses the source code itself within machine learning heavily drawing inspiration from existing machine-learning methods in NLP and elsewhere. The third wave promises new machine-learning models informed by programming language semantics. What form will it take? At the time of this writing, machine learning, and deep learning in particular, is enjoying rockstar status among research fields. Despite its current (perhaps ephemeral) popularity, it is not a panacea. In some cases, a machine-learning model may not be required (e.g., when the problem is deterministic), and, in other cases, simple models can outperform advanced, off-the-self deep-learning methods, designed for non-code domains BIB015 BIB016 . Furthermore, over-engineering or under-engineering machine-learning models usually leads to suboptimal results. Selecting a machine-learning model for a specific problem necessitates questioning if a specific model is fit for the target application. Strong baseline methods are needed to estimate if a performance improvement justifies the added complexity of a model. In short, the right tools should be used for the right job, and machine learning is no exception. Bridging Representations and Communities. Programming language research and practice use a well-defined and widely useful set of representations, usually in a symbolic form. In contrast, machine-learning research customarily works with continuous representations. Bridging the gap between these representations by allowing machine-learning systems to reason with programming language representations is an important challenge. Handling unambiguous code has already led to a combination of probabilistic methods with "formal" constraints that limit the probabilistic model to valid code. For example, Maddison and Tarlow BIB005 limit their model to generate syntactically correct and scope-respecting code, while Raychev et al. BIB006 easily create a highly structured model of a program taking advantage of its unambiguous form. Introducing better representations that bridge the gap between machine learning and source code will allow the probabilistic models of code to reason about the rich structure and semantics of code, without resorting to proxies. The core problem in this area is the lack of understanding of machine learning from the programming language community and vice versa. At the same time, new machine-learning methods that can handle programming language structures in its full complexity at scale need to be researched. A major obstacle here is engineering systems that efficiently and effectively combine the probabilistic world of machine learning and the formal, logic-based world of code analysis. One approach, taken by several authors BIB008 BIB007 BIB006 BIB017 , is to relax formal systems into probabilistic. Such systems, however, lack the guarantees formal systems (e.g., soundness) often provide. Alemi et al. BIB009 , Balog et al. BIB010 , and Loos et al. BIB018 follow a second approach that maintains soundness: They learn input and context-specific heuristics that efficiently guide search-based methods, such as theorem proving and program synthesis. All of approaches to source code modeling must decide whether to explicitly model the structure and constraints of source code or to rely on general methods with adequate capacity. One one hand, using well-known, domain-generic machine-learning methods has the advantage that the models are well understood and rarely require significant effort or expertise to apply. On the other hand, designing models with built-in inductive biases for the problem domain usually performs better with fewer data, at the cost of manually designing and debugging domain (even problemspecific) networks. One such promising direction is modular neural network architectures. Such architectures decompose the network into components that are combined based on the problem instance. For source code models, such architectures can derive its structure through static analyses. These architectures have been useful for visual question answering in NLP. Andreas et al. BIB011 create neural networks by composing them from "neural modules," based on the input query structure. Similarly, we believe that such architectures will be useful within probabilistic models of source code. An early example is the work of Allamanis et al. BIB019 , who design a neural network based on the output of data flow analysis. Such architectures should not only be effective for bridging representations among communities but also-as we will discuss next-can combat issues with compositionality, sparsity, and generalization. Nevertheless, issues that arise in static analyses, such as path explosion, will still need to be addressed. Data Sparsity, Compositionality, and Strong Generalization. The principle of reusability in software engineering creates a form of sparsity in the data, where it is rare to find multiple source code elements that perform exactly the same tasks. For example, it is rare to find hundreds of database systems, whereas one can easily find thousands of news articles on a popular piece of news. The exceptions, like programming competitions and student solutions to programming assignments, are quite different from industrial code. This suggests that there are many opportunities for researching machine-learning models and inference methods that can handle and generalize from the highly structured, sparse, and composable nature of source code data. Do we believe in the unreasonable effectiveness of data BIB003 ? Yes, but we do not have sufficient data. Although code and text are both intrinsically extensible, code pushes the limit of existing machine-learning methods in terms of representing composition. This is because most natural language methods rarely define novel, previously unseen, terms, with the possible exception of legal and scientific texts. In contrast, source code is inherently extensible, with developers constantly creating new terms (e.g., by defining new functions and classes) and combining them in still higher-level concepts. Compositionality refers to the idea that the meaning of some element can be understood by composing the meaning of its constituent parts. Recent work BIB012 has shown that deep-learning architecture can learn some aspects of compositionality in text. Machine learning for highly compositional objects remains challenging, because it has proven hard to capture relations between objects, especially across abstraction levels. Such challenges arise even when considering simple codelike expressions BIB008 . However, if sufficient progress is to be made, then representing source code artifacts in machine learning will improve significantly, positively affecting other downstream tasks. For example, learning composable models that can combine meaningful representations of variables into meaningful representations of expressions and functions will lead to much stronger generalization performance. Data sparsity is still an important and unsolved problem. Although finding a reasonably large amount of source code is relatively easy, it is increasingly hard to retrieve some representations of source code. Indeed, it is infeasible even to compile all of the projects in a corpus of thousands of projects, because compiling a project requires understanding how the project handles external dependencies, which can sometimes be idiosyncratic. Furthermore, computing or acquiring semantic properties of existing, real-world code (e.g., purity of a function BIB002 or pre-/post-conditions BIB001 ) is hard to do, especially at scale. Scalability also hampers harvesting run-time data from real-world programs: It is challenging to acquire substantial run-time data even for a single project. Exploring ways to synthesize or transform "natural" programs that perform the same task in different ways is a possible way ahead. Another promising direction to tackle this issue is by learning to extrapolate from run-time data (e.g., collected via instrumentation of a test-suite) to static properties of the code  . Although this is inherently a noisy process, achieving high accuracy may be sufficient, thanks to the inherent ability of machine learning to handle small amounts of noise. Strong generalization also manifests as a deployability problem. Machine-learning models, especially when they have become effective, are often so large that they are too large for a developer's machine, but using the cloud raises privacy 6 concerns and prevents offline coding. When under development and tooling is needed, code evolves quickly, subjecting models to constant concept drift and necessitating frequent retraining, which can be extremely slow and costly. Addressing this deployability concern is an open problem and requires advances in machine-learning areas such as transfer learning and one-shot learning. For example, say a program P uses libraries A and B, which have been shipped with the models M A and M B . Could we save time training a model for P by transferring knowledge from M A and M B ? Finally, source code representations are multifaceted. For example, the token-level "view" ofsource code is quite different from a data flow view of code. Learning to exploit multiple views simultaneously can help machine-learning models generalize and tackle issues with data sparsity. Multi-view BIB004 and multi-modal learning (e.g., Gella et al. BIB013 ), areas actively explored in machine learning, aim to achieve exactly this. By combining multiple representations of data, they aim to improve on the performance on various tasks, learning to generalize using multiple input signals. We believe that this is a promising future direction that may allow us to combine probabilistic representations of code to achieve better generalization. Measures. To train and evaluate machine-learning models, we need to easily measure their performance. These measures allow the direct comparison of models and have already lead to improvements in multiple areas, such as code completion (Section 5.1). Nonetheless, these measures are imprecise. For instance, probabilistic recommender systems define a probability density over suggestions whose cross-entropy can be computed against the empirical distribution in test data. Although cross-entropy is correlated with suggestion accuracy and confidence, small improvements in cross-entropy may not improve accuracy. Sometimes the imprecision is due to unrealistic use case assumptions. For example, the measures for LM-based code completion tend to assume that code is written sequentially, from the first token to the last one. However, developers rarely write code in such a simple and consistent way BIB014 . Context-based approaches assume that the available context (e.g., other object usages in the context) is abundant, which is not true in a real editing scenarios. Researchers reporting keystrokes saved have usually assumed that code completion suggestions are continuously presented to the user as he or she is typing. When the top suggestion is the target token, the user presses a single key (e.g., return) to complete the rest of the target. Furthermore, some metrics that are widely used in NLP are not suited for source code. For example, BLEU score is not suitable for measuring the quality of output source code (e.g., in transducer models), because it fails to account for the fact that the syntax is known in all programming languages, so the BLEU score may be artificially "inflated" for predicting deterministic syntax. Second, the granularity over which BLEU is computed (e.g., per-statement vs. per-token) is controversial. Finally, syntactically diverse answers may be semantically equivalent, yielding a low BLEU score while being correct. Finding new widely accepted measures for various tasks will allow the community to build reliable models with well-understood performance characteristics.