The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers <s> Performance Analysis <s> This paper presents a novel approach to learning to solve simple arithmetic word problems. Our system, ARIS, analyzes each of the sentences in the problem statement to identify the relevant variables and their values. ARIS then maps this information into an equation that represents the problem, and enables its (trivial) solution as shown in Figure 1. The paper analyzes the arithmetic-word problems “genre”, identifying seven categories of verbs used in such problems. ARIS learns to categorize verbs with 81.2% accuracy, and is able to solve 77.7% of the problems in a corpus of standard primary school test questions. We report the first learning results on this task without reliance on predefined templates and make our data publicly available. 1 <s> BIB001 </s> The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers <s> Performance Analysis <s> This paper describes our system which solves simple arithmetic word problems. The system takes a word problem described in natural language, extracts information required for representation, orders the facts presented, applies procedures and derives the answer. It then displays this answer in natural language. The emphasis of this paper is on the natural language processing(NLP) techniques used to retrieve the relevant information from the English word problem. The system shows improvements over existing systems. <s> BIB002 </s> The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers <s> Performance Analysis <s> This paper formalizes the problem of solving multi-sentence algebraic word problems as that of generating and scoring equation trees. We use integer linear programming to generate equation trees and score their likelihood by learning local and global discriminative models. These models are trained on a small set of word problems and their answers, without any manual annotation, in order to choose the equation that best matches the problem text. We refer to the overall system as ALGES. We compare ALGES with previous work and show that it covers the full gamut of arithmetic operations whereas Hosseini et al. (2014) only handle addition and subtraction. In addition, ALGES overcomes the brittleness of the Kushman et al. (2014) approach on single-equation problems, yielding a 15% to 50% reduction in error. <s> BIB003 </s> The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers <s> Performance Analysis <s> Solving simple arithmetic word problems is one of the challenges in Natural Language Understanding. This paper presents a novel method to learn to use formulas to solve simple arithmetic word problems. Our system, analyzes each of the sentences to identify the variables and their attributes; and automatically maps this information into a higher level representation. It then uses that representation to recognize the presence of a formula along with its associated variables. An equation is then generated from the formal description of the formula. In the training phase, it learns to score the pair from the systematically generated higher level representation. It is able to solve 86.07% of the problems in a corpus of standard primary school test questions and beats the state-of-the-art by <s> BIB004 </s> The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers <s> Performance Analysis <s> This demonstration presents a tag-based statistical English math word problem (MWP) solver with understanding, reasoning, and explanation. It analyzes the text and transforms both body and question parts into their tag-based logic forms, and then performs inference on them. The proposed tag (e.g., Agent, Verb, etc.) provides the flexibility for annotating an extracted math quantity with its associated syntactic and semantic information, which can be used to identify the desired operand and filter out irrelevant quantities (so that the answer can be obtained precisely). Since the physical meaning of each quantity is explicitly represented with those tags and used in the inference process, the proposed approach could explain how the answer is obtained in a human comprehensible way. <s> BIB005 </s> The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers <s> Performance Analysis <s> This paper presents a novel approach to automatically solving arithmetic word problems. This is the first algorithmic approach that can handle arithmetic problems with multiple steps and operations, without depending on additional annotations or predefined templates. We develop a theory for expression trees that can be used to represent and evaluate the target arithmetic expressions; we use it to uniquely decompose the target arithmetic problem to multiple classification problems; we then compose an expression tree, combining these with world knowledge through a constrained inference framework. Our classifiers gain from the use of quantity schemas that supports better extraction of features. Experimental results show that our method outperforms existing systems, achieving state of the art performance on benchmark datasets of arithmetic word problems. <s> BIB006 </s> The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers <s> Performance Analysis <s> Math word problems provide a natural abstraction to a range of natural language understanding problems that involve reasoning about quantities, such as interpreting election results, news about casualties, and the financial section of a newspaper. Units associated with the quantities often provide information that is essential to support this reasoning. This paper proposes a principled way to capture and reason about units and shows how it can benefit an arithmetic word problem solver. This paper presents the concept of Unit Dependency Graphs (UDGs), which provides a compact representation of the dependencies between units of numbers mentioned in a given problem. Inducing the UDG alleviates the brittleness of the unit extraction system and allows for a natural way to leverage domain knowledge about unit compatibility, for word problem solving. We introduce a decomposed model for inducing UDGs with minimal additional annotations, and use it to augment the expressions used in the arithmetic word problem solver of (Roy and Roth 2015) via a constrained inference framework. We show that introduction of UDGs reduces the error of the solver by over 10 %, surpassing all existing systems for solving arithmetic word problems. In addition, it also makes the system more robust to adaptation to new vocabulary and equation forms . <s> BIB007
Given the aforementioned datasets, we merge the experimental results reported from previous works into one table. Such a unified organization can facilitate readers in identifying the methods with superior performance in each dataset. As shown in Table 2 , the rows refer to the corpus of datasets and the columns are the statistic-based and tree-based methods. The cells are filled with the accuracies of these algorithms when solving math word problems in different datasets. We conduct additional experiments to cover all the cells by the tree-based solutions. These new experiment results are highlighted in blue color. Those with missing value are indicated by "-" and it means that there was no experiment conducted for the algorithm in the particular dataset. The main reason is that they require particular efforts on logic templates and annotations, which are very trivial and cumbersome for experiment reproduction. There is no algorithm comparison for the dataset Math23K because the problem text is in Chinese and the feature extraction technologies proposed in the statistic-based and tree-based approaches are not applicable. From the results in Table 2 , we derive the following observations worth noting and provide reasonings to explain the results. First, the statistic-based methods with advanced logic representation, such as Schema BIB002 , Formula BIB004 and Logic-Form  , BIB005 , achieve dominating performance in the AI2 dataset. Their superiority is primarily owned to the additional efforts on annotating the text problem with more advanced logic representation. These annotations allow them to conduct finegrained reasoning. In contrast, ARIS BIB001 does not work as good because it focuses on "change" schema of quantities and does not fully exploit other schemas like "compare" BIB002 . Since there are only hundreds of math problems in the datasets, it is feasible to make an exhaustive scan on the math problems and manually define the templates to fit these datasets. For instance, all quantities and the main-goal are first identified by rules in LogicForm  , BIB005 and explicitly associated with their roletags. Thus, with sufficient human intervention, the accuracy of statistic-baesd methods in AI2 can boost to 88.64%, much higher than that of tree-based methods. Nevertheless, these statistic-based methods are considered as brittle and rigid  and not scalable to handle large and diversified datasets, primarily due to the heavy annotation cost to train an accurate mapping between the text and the logic representation. Second, the results of tree-based methods in AI2, IL and CC are collected from  where the same experimental setting of 3fold cross validation is applied. It is interesting to observe that ALGES BIB003 , ExpressionTree BIB006 and UNITDEP BIB007 cannot perform equally well on the three datasets. ALGES works poorly in AI2 because irrelevant quantities exist in its math problems and ALGES is not trained with a classifier to get rid of them. However, it outperforms ExpressionTree and UNITDEP by a wide As to the datasets of SingleEQ and AllArith, UNITDEP is a winner in both datasets, owning to the effectiveness of the proposed unit dependency graph (UDG). In the math problems with operators {×, ÷}, the unit and rate are important clues to determine the correct quantities and operators in the math expression. The UDG poses constraints on unit compatibility to filter the false candidates in the expression tree construction. It can alleviate the brittleness of the unit extraction system, even though it requires additional annotation overhead in order to induce UDGs. Last but not the least, these experiments were conducted on small-scale datasets and their performances on larger and more diversified datasets remain unclear. Recently, Huang et al. have noticed the gap and released Dolphin18K  which contains 18, 460 problems and 5, 871 templates with one or multiple equations. The findings in  are astonishing. The accuracies of existing approaches for equation set problems, which will be introduced in the next section, degrade sharply to less than 25%. These methods cannot even perform better than a simple baseline that first uses text similarity to find the most similar text problem in the training dataset and then fills the number slots in its associated equation template. To discover the performance of existing arithmetic problem solvers in large-scale datasets, we conduct similar experiments to examine the performance of tree-based approaches in Dolphin-S. The results are shown in the last two columns of Table 2 , leading to two conclusions. First, the overall performances of existing tree-based arithmetic problem solvers are not promising in large-scale and diversified datasets. There is still great room for improvement in the research area of arithmetic math word problem solver. Second, MathDQN exhibits superior performance over its two tree-based competitors, verifying its robustness by using the deep reinforcement learning framework. The performances of the equation set solvers on the existing datasets are reported in Table 4 . From the experimental results, we derive the following observations and discussions. First, there are many empty cells in the table. In the ideal case, the algorithms should be conducted on all the available benchmark datasets and compared with all the previous competitors. The reasons are multi-fold, such as limitation of the implementation (e.g., as Upadhyay stated in  , they could not run ZDC on DRAW1K because ZDC can only handle limited types of equation systems), delayed release of the dataset (e.g., the Dolphin18K dataset has not been released when the work of DNS is published) or unfitness in certain scenarios (e.g., the experiments of FG-Expression  were only conducted on Dolphin18K because the authors considered that the previous datasets are not suitable due to their limitation in scalability and diversity). It is noticeable that such an incomplete picture brings difficulty to judge the performance and may miss certain insightful findings. Second, ALG514 is the smallest dataset and also the most widely adopted dataset for performance evaluation. Among the template-based methods, MixedSP outperforms KAZB and ZDC because it benefits from the mined implicit supervision from an external source of additional 2, 000 samples. As reported in  , if only the explicit dataset (i.e., the problems in ALG514) is used, its performance is slightly inferior to ZDC. A possible reason to explain this is that ZDC uses a richer set of features based on POS tags, coreference and dependency parses. In contrast, MixedSP only uses features based on POS tags. It is also interesting to see that SIM and DNS obtain the same accuracy on ALG514. This is the dataset is too small to train an effective deep learning model. The reported accuracy of seq2seq model is only 16.1% in ALG514. DNS is a hybrid approach that combines a seq2seq model and similarity retrieval model. It means the deep learning 1. https://answers.yahoo.com/ model does not take any effect when handling problems in ALG514. Finally, the datasets of Dolphin1878 and DRAW1K are released for the approaches of DOL and MixedSP, respectively. In the experimental settings, simple baselines such as SIM based on textual similarity or KAZB which is the earliest template-based method, are selected. It is not surprising to see that Dolphin1878 and DRAW1K outperform their competitors by a large margin. Nevertheless, the research for equation set solvers has shifted to proposing methods that can work well in a large and diversified dataset such as Dolphin18K. We implemented our own version of DNS and evaluated its performance on the large dataset. Unfortunately, we did not observe a higher accuracy derived from DNS. The reasons could be two-fold. First, our implementation may not be optimized and the model parameters may not be well tuned. Second, there are thousands of templates in the datasets, which may bring challenges for the classification task. Nevertheless, with the availability of large-scale datasets, applying deep learning models for MWPs is still an interesting research direction that deserves intensive attention.