The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers <s> INTRODUCTION <s> A program was written to solve calculus word problems. The program, CARPS (CAlculus Rate Problem Solver), is restricted to rate problems. The overall plan of the program is similar to Bobrow's STUDENT, the primary difference being the introduction of "structures" as the internal model in CARPS. Structures are stored internally as trees, each structure holding the information gathered about one object. ::: ::: It was found that the use of structures made CARPS more powerful than STUDENT in several respects. ::: ::: 1) CARPS is better able to recognize that two phrases describe the same object. (e.g., CONICAL PILE and PILE OF SAND) 2) Information about an object can be gathered in piecemeal fashion. In calculus word problems it is not uncommon to have two or three sentences providing information for one equation. 3) CARPS to a limited degree is able to use its knowledge to parse its input sentences. For example, in a problem about a filter, ALTITUDE was interpreted as ALTITUDE OF THE FILTER because CARPS knew that since the filter was a cone and cones have altitudes the filter had an altitude. ::: ::: The program has solved 14 calculus problems, most taken (sometimes with slight modifications) from standard calculus texts. <s> BIB001 </s> The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers <s> INTRODUCTION <s> In this paper, I describe a computer program, WORDPRO, which simulates the psychological processes involved when third-grade children understand and solve simple arithmetic word problems. Both the implementation of the program and its performance on a set of sample problems are presented. WORDPRO is a useful research tool in that it demonstrates the sufficiency of the theory upon which it is based, assists in communicating that theory to other researchers, and provides a sources of empirical predictions for experimental tests of the theory. <s> BIB002 </s> The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers <s> INTRODUCTION <s> Understanding of free-format multi-step arithmetic word problems with extraneous information is discussed. A model including a full set of general skills necessary for understanding such problems was developed and computer implemented. The validity of the simulation was confirmed by testing on a variety of word problems with extraneous information. The results of this study show the way for moving from an intuitive understanding of word problems to a conscious and controlled process. <s> BIB003 </s> The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers <s> INTRODUCTION <s> This article addresses the problem of understanding mathematics described in natural language. Research in this area dates back to early 1960s. Several systems have so far been proposed to involve machines to solve mathematical problems of various domains like algebra, geometry, physics, mechanics, etc. This correspondence provides a state of the art technical review of these systems and approaches proposed by different research groups. A unified architecture that has been used in most of these approaches is identified and differences among the systems are highlighted. Significant achievements of each method are pointed out. Major strengths and weaknesses of the approaches are also discussed. Finally, present efforts and future trends in this research area are presented. <s> BIB004 </s> The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers <s> INTRODUCTION <s> The explosion of image data on the Internet has the potential to foster more sophisticated and robust models and algorithms to index, retrieve, organize and interact with images and multimedia data. But exactly how such data can be harnessed and organized remains a critical problem. We introduce here a new database called “ImageNet”, a large-scale ontology of images built upon the backbone of the WordNet structure. ImageNet aims to populate the majority of the 80,000 synsets of WordNet with an average of 500-1000 clean and full resolution images. This will result in tens of millions of annotated images organized by the semantic hierarchy of WordNet. This paper offers a detailed analysis of ImageNet in its current state: 12 subtrees with 5247 synsets and 3.2 million images in total. We show that ImageNet is much larger in scale and diversity and much more accurate than the current image datasets. Constructing such a large-scale database is a challenging task. We describe the data collection scheme with Amazon Mechanical Turk. Lastly, we illustrate the usefulness of ImageNet through three simple applications in object recognition, image classification and automatic object clustering. We hope that the scale, accuracy, diversity and hierarchical structure of ImageNet can offer unparalleled opportunities to researchers in the computer vision community and beyond. <s> BIB005 </s> The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers <s> INTRODUCTION <s> Machine learning is traditionally formalized and researched as the study of learning concepts and decision functions from labeled examples, requiring a representation that encodes information about the domain of the decision function to be learned. We are interested in providing a way for a human teacher to interact with an automated learner using natural instructions, thus allowing the teacher to communicate the relevant domain expertise to the learner without necessarily knowing anything about the internal representations used in the learning process. ::: ::: In this paper we suggest to view the process of learning a decision function as a natural language lesson interpretation problem instead of learning from labeled examples. This interpretation of machine learning is motivated by human learning processes, in which the learner is given a lesson describing the target concept directly, and a few instances exemplifying it. We introduce a learning algorithm for the lesson interpretation problem that gets feedback from its performance on the final task, while learning jointly (1) how to interpret the lesson and (2) how to use this interpretation to do well on the final task. This approach alleviates the supervision burden of traditional machine learning by focusing on supplying the learner with only human-level task expertise for learning. ::: ::: We evaluate our approach by applying it to the rules of the Freecell solitaire card game. We show that our learning approach can eventually use natural language instructions to learn the target concept and play the game legally. Furthermore, we show that the learned semantic interpreter also generalizes to previously unseen instructions. <s> BIB006 </s> The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers <s> INTRODUCTION <s> We consider the challenge of learning semantic parsers that scale to large, open-domain problems, such as question answering with Freebase. In such settings, the sentences cover a wide variety of topics and include many phrases whose meaning is difficult to represent in a fixed target ontology. For example, even simple phrases such as ‘daughter’ and ‘number of people living in’ cannot be directly represented in Freebase, whose ontology instead encodes facts about gender, parenthood, and population. In this paper, we introduce a new semantic parsing approach that learns to resolve such ontological mismatches. The parser is learned from question-answer pairs, uses a probabilistic CCG to build linguistically motivated logicalform meaning representations, and includes an ontology matching model that adapts the output logical forms for each target ontology. Experiments demonstrate state-of-the-art performance on two benchmark semantic parsing datasets, including a nine point accuracy improvement on a recent Freebase QA corpus. <s> BIB007 </s> The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers <s> INTRODUCTION <s> While there has been an explosion of impressive, data-driven AI applications in recent years, machines still largely lack a deeper understanding of the world to answer questions that go beyond information explicitly stated in text, and to explain and discuss those answers. To reach this next generation of AI applications, it is imperative to make faster progress in areas of knowledge, modeling, reasoning, and language. Standardized tests have often been proposed as a driver for such progress, with good reason: Many of the questions require sophisticated understanding of both language and the world, pushing the boundaries of AI, while other questions are easier, supporting incremental progress. In Project Aristo at the Allen Institute for AI, we are working on a specific version of this challenge, namely having the computer pass Elementary School Science and Math exams. Even at this level there is a rich variety of problems and question types, the most difficult requiring significant progress in AI. Here we propose this task as a challenge problem for the community, and are providing supporting datasets. Solutions to many of these problems would have a major impact on the field so we encourage you: Take the Aristo Challenge! <s> BIB008 </s> The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers <s> INTRODUCTION <s> We propose the task of free-form and open-ended Visual Question Answering (VQA). Given an image and a natural language question about the image, the task is to provide an accurate natural language answer. Mirroring real-world scenarios, such as helping the visually impaired, both the questions and answers are open-ended. Visual questions selectively target different areas of an image, including background details and underlying context. As a result, a system that succeeds at VQA typically needs a more detailed understanding of the image and complex reasoning than a system producing generic image captions. Moreover, VQA is amenable to automatic evaluation, since many open-ended answers contain only a few words or a closed set of answers that can be provided in a multiple-choice format. We provide a dataset containing ~0.25M images, ~0.76M questions, and ~10M answers (www.visualqa.org), and discuss the information it provides. Numerous baselines for VQA are provided and compared with human performance. <s> BIB009 </s> The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers <s> INTRODUCTION <s> Given the well-known limitations of the Turing Test, there is a need for objective tests to both focus attention on, and measure progress towards, the goals of AI. In this paper we argue that machine performance on standardized tests should be a key component of any new measure of AI, because attaining a high level of performance requires solving significant AI problems involving language understanding and world modeling - critical skills for any machine that lays claim to intelligence. In addition, standardized tests have all the basic requirements of a practical test: they are accessible, easily comprehensible, clearly measurable, and offer a graduated progression from simple tasks to those requiring deep understanding of the world. Here we propose this task as a challenge problem for the community, summarize our state-of-the-art results on math and science tests, and provide supporting datasets <s> BIB010 </s> The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers <s> INTRODUCTION <s> Problems at the intersection of vision and language are of significant importance both as challenging research questions and for the rich set of applications they enable. However, inherent structure in our world and bias in our language tend to be a simpler signal for learning than visual modalities, resulting in models that ignore visual information, leading to an inflated sense of their capability. ::: We propose to counter these language priors for the task of Visual Question Answering (VQA) and make vision (the V in VQA) matter! Specifically, we balance the popular VQA dataset by collecting complementary images such that every question in our balanced dataset is associated with not just a single image, but rather a pair of similar images that result in two different answers to the question. Our dataset is by construction more balanced than the original VQA dataset and has approximately twice the number of image-question pairs. Our complete balanced dataset is publicly available at this http URL as part of the 2nd iteration of the Visual Question Answering Dataset and Challenge (VQA v2.0). ::: We further benchmark a number of state-of-art VQA models on our balanced dataset. All models perform significantly worse on our balanced dataset, suggesting that these models have indeed learned to exploit language priors. This finding provides the first concrete empirical evidence for what seems to be a qualitative sense among practitioners. ::: Finally, our data collection protocol for identifying complementary images enables us to develop a novel interpretable model, which in addition to providing an answer to the given (image, question) pair, also provides a counter-example based explanation. Specifically, it identifies an image that is similar to the original image, but it believes has a different answer to the same question. This can help in building trust for machines among their users. <s> BIB011
Designing an automatic solver for mathematical word problems (MWPs) has a long history dated back to the 1960s  ,  , BIB001 , and continues to attract intensive research attention. In the past three years, more than 40 publications on this topic have emerged in the premier venues of artificial intelligence. The problem is particularly challenging because there remains a wide semantic gap to parse the human-readable words into machineunderstandable logics so as to facilitate quantitative reasoning. Hence, MWPs solvers are broadly considered as good test beds to evaluate the intelligence level of agents in terms of natural language understanding BIB008 , BIB010 and the successful solving of MWPs would constitute a milestone towards general AI. We categorize the evolving of MWP solvers into three major stages according to the technologies behind these solvers, as depicted in Figure 1 . In the first pioneering stage, roughly from the year 1960 to 2010, systems such as STUDENT  , DEDUCOM  , WORDPRO BIB002 and ROBUST BIB003 , manually craft rules and schemas for pattern matchings. Thereupon, these solvers heavily rely on human interventions and can only resolve a limited number of scenarios that are defined in advance. Those early efforts for automatic understanding of natural language mathematical problems have been thoroughly reviewed in BIB004 . We exclude them from the scope of this survey paper and focus on the recent technology developments that have not been covered in the previous survey BIB004 . In the second stage, MWP solvers made use of semantic parsing BIB006 , BIB007 , with the objective of mapping the sentences from problem statements into structured logic representations so as to facilitate quantitative reasoning. It has regained considerable • D. Zhang interests from the academic, and a booming number of methods have been proposed in the past years. These methods leveraged various strategies of feature engineering and statistical learning for performance boosting. The authors of these methods also claimed promising results in their public or manually harvested datasets. In this paper, one of our tasks is to present a comprehensive review on the proposed methods in this stage. The methods will be first organized according to the sub-tasks of MWPs which they were designed to solve, such as arithmetic word problem (in Section 2), equation set word problem (in Section 3) and geometric word problem (in Section 5). We then examine the proposed techniques in each sub-task with a clear technical organization and accountable experimental evaluations. MWP solvers in the third stage originated from an empirical work  deserve some special attention. Its experimental results on a large-scale and diversified dataset showed that the status of MWP solvers was not as optimistic as they claimed to be. In fact, the accuracies of many approaches dropped sharply and there is a great room for improvement in this research area. To design more accurate and robust solutions, the subsequent publications are forked into two directions. One is to continue refining the technology of semantic parsing. For instance, Huang et al. proposed a new type of semantic representation to conduct finegrained inference  . The other direction attempts to exploit the advantages of deep learning models, with the availability of largescale training datasets. This is an emerging research direction for MWP solvers and we observed two instances, including Deep Neural Solver  using recurrent neural network to generate the equation template, and MathDQN  relying on the deep reinforcement learning framework. It is expected to witness more and more deep learning based methods to solve MWPs in the future. To sum up, we present a comprehensive survey to review the MWP solvers proposed in recent years. Researchers in the community can benefit from this survey in the following ways: 1) We provide a wide coverage on the math word problems, including arithmetic word problem, equation set problem, geometry word problem and miscellaneous sub-tasks related to automatic math solvers. The practicers can easily identify all relevant approaches for performance evaluations. We observed that the unawareness of relevant competitors is not uncommon in the past literature. We are positive that the availability of our survey can help avoid such unawareness. 2) The solvers designed for arithmetic word problems (AWP) with only one unknown variable and equation set problems (ESP) with multiple unknown variables are often not differentiated by previous works. In fact, the methods proposed for ESP are more general and can be used to solve AWP. In this survey, we clearly identify the difference and organize them in separate sections. We also generalize their methods in terms of the number of operands and the type of operators they are capable to support. 3) Feature engineering plays an vital role to bridge the gap of semantic parsing. Almost all MWP solvers state their strategies of crafting effective features, resulting in a very diversified group of features, and there is a lack of clear organization among these features. In this survey, we will be the first to summarize all these proposed features in Table 5 . 4) As for the fairness on performance evaluations, ideally, there should be a benchmark dataset well accepted and widely adopted by the MWP research community, just like Ima-geNet BIB005 for visual object recognition and VQA BIB009 , BIB011 for visual question answering. Unfortunately, we observed that many approaches tend to compile their own datasets to verify their superiorities, which result in missing of relevant competitors as mentioned above. Tables 2 and 3 integrate the results of existing methods on all the public datasets. After collecting the accuracies that have been reported in the past literature, we observed many empty cells in the table. Each empty cell refers to a missing experiment on a particular algorithm and dataset. In this survey, we make our best efforts to fill the missing results by conducting a considerable number of additional experiments, and guarantee that this survey provides comprehensive comparison and delivers explicit experimental analysis. The remainder of the paper is organized as follows. We first review the arithmetic word problem solvers in Section 2, followed by equation set problem solvers in Section 3. Since feature engineering deserves special attention, we summarize the extracted features as well as the associated pre-processing techniques in Section 4. The geometric word problem solvers are reviewed in Section 5. We also cover miscellaneous automatic solvers related to math problems in Section 6. We conclude the paper and point out several future directions in MWPs that are worth examination in the final section.