The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers <s> Benchmark Datasets <s> We present an approach for automatically learning to solve algebra word problems. Our algorithm reasons across sentence boundaries to construct and solve a system of linear equations, while simultaneously recovering an alignment of the variables and numbers in these equations to the problem text. The learning algorithm uses varied supervision, including either full equations or just the final answers. We evaluate performance on a newly gathered corpus of algebra word problems, demonstrating that the system can correctly answer almost 70% of the questions in the dataset. This is, to our knowledge, the first learning result for this task. <s> BIB001 </s> The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers <s> Benchmark Datasets <s> This paper presents a semantic parsing and reasoning approach to automatically solving math word problems. A new meaning representation language is designed to bridge natural language text and math expressions. A CFG parser is implemented based on 9,600 semi-automatically created grammar rules. We conduct experiments on a test set of over 1,500 number word problems (i.e., verbally expressed number problems) and yield 95.4% precision and 60.2% recall. <s> BIB002 </s> The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers <s> Benchmark Datasets <s> We propose a new evaluation for automatic solvers for algebra word problems, which can identify mistakes that existing evaluations overlook. Our proposal is to evaluate such solvers using derivations, which reflect how an equation system was constructed from the word problem. To accomplish this, we develop an algorithm for checking the equivalence between two derivations, and show how derivation an- notations can be semi-automatically added to existing datasets. To make our experiments more comprehensive, we include the derivation annotation for DRAW-1K, a new dataset containing 1000 general algebra word problems. In our experiments, we found that the annotated derivations enable a more accurate evaluation of automatic solvers than previously used metrics. We release derivation annotations for over 2300 algebra word problems for future evaluations. <s> BIB003
There have been four datasets specifically collected for the equation set problems that involve multiple unknown variables. We present their descriptions in the following and summarize the statistics of the datasets in Table 3 . We use # of problems # of templates to report the average number of problems associated with each template. We noticed that in each dataset, a small fraction of problems are associated with one unknown variable in the template. Thus, we also report the number of single-equation problems in each dataset. 1) ALG514 BIB001 . The dataset is crawled from Algebra.com, a crowd-sourced tutoring website. The problems are posted by students. The problems with information gap or require explicit background knowledge are discarded. Consequently, a set of 1024 questions is collected and cleaned by crowdworkers in Amazon Mechanical Turk. These problems are further filtered as the authors require each equation template to appear for at least 6 times. Finally, 514 problems are left in the dataset. 2) Dolphin1878 BIB002 . Its math problems are crawled from two websites: albegra.com and answers.yahoo.com. For math problems in answers.yahoo.com, the math equations and answers are manually added by human annotators. Finally, the dataset combined from the two sources contains 1, 878 math problems with 1183 equation templates. 3) DRAW1K BIB003 . The authors of DRAW1K argued that Dolphin1878 has limited textual variations and lacks narrative. This motivated them to construct a new dataset that is diversified in both vocabularies and equation systems. With these two objectives, they constructed DRAW1K with exactly 1, 000 linear equation problems that are crawled and filtered from algebra.com. 4) Dolphin18K  . The dataset is collected and rectified mainly from the math category of Yahoo! Answers 1 . The problems, equation system annotations, and answers are extracted semi-automatically, with great intervention of human efforts. The procedure consists of four stages: removing irrelevant problems, cleaning problem text, extracting gold answers and constructing equation system annotations. The harvested dataset is so far the largest one, with 18, 460 problems and 5, 871 equation templates.