The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers <s> Text-Aligned Diagram Understanding <s> We describe a program, BEATRIX, that can understand textbook physics problems specified by a combination of English text and a diagram. The result of the understanding process is a unified internal model that represents the problem, including information derived from both the English text and the diagram. The system is implemented as two opportunistic coparsers, one for English and one for diagrams, within a blackboard architecture. A central problem is establishing coreference, that is, determining when parts of the text and diagram refer to the same object. Constraints supplied by the text and diagram mutually reduce ambiguity in interpretation of the other modality. <s> BIB001 </s> The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers <s> Text-Aligned Diagram Understanding <s> Pattern information and natural language information used together can complement and reinforce each other to enable more effective communication than can either medium alone (Feiner 91) (Nakamura 93). One of the good examples is a pictorial book of flora (PBF) . In the PBF, readable explanations which combine texts, pictures, and diagrams are achieved, as shown in Figure 1 and 2. Taking advantage of this, we propose here a new method for analyzing the PBF diagrams by using natural language information. In the PBF, many kinds of information about plants are generally stored in the following media: <s> BIB002 </s> The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers <s> Text-Aligned Diagram Understanding <s> This paper surveys research in developing computational models for integrating linguistic and visual information. It begins with a discussion of systems which have been actually implemented and continues with computationally motivated theories of human cognition. Since existing research spans several disciplines (e.g., natural language understanding, computer vision, knowledge representation), as well as several application areas, an important contribution of this paper is to categorize existing research based on inputs and objectives. Finally, some key issues related to integrating information from two such diverse sources are outlined and related to existing research. Throughout, the key issue addressed is the correspondence problem, namely how to associate visual events with words and vice versa. <s> BIB003 </s> The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers <s> Text-Aligned Diagram Understanding <s> Automatically solving geometry questions is a long-standing AI problem. A geometry question typically includes a textual description accompanied by a diagram. The first step in solving geometry questions is diagram understanding, which consists of identifying visual elements in the diagram, their locations, their geometric properties, and aligning them to corresponding textual descriptions. In this paper, we present a method for diagram understanding that identifies visual elements in a diagram while maximizing agreement between textual and visual data. We show that the method's objective function is submodular; thus we are able to introduce an efficient method for diagram understanding that is close to optimal. To empirically evaluate our method, we compile a new dataset of geometry questions (textual descriptions and diagrams) and compare with baselines that utilize standard vision techniques. Our experimental evaluation shows an F1 boost of more than 17% in identifying visual elements and 25% in aligning visual elements with their textual descriptions. <s> BIB004 </s> The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers <s> Text-Aligned Diagram Understanding <s> Diagrams are common tools for representing complex concepts, relationships and events, often when it would be difficult to portray the same information with natural images. Understanding natural images has been extensively studied in computer vision, while diagram understanding has received little attention. In this paper, we study the problem of diagram interpretation, the challenging task of identifying the structure of a diagram and the semantics of its constituents and their relationships. We introduce Diagram Parse Graphs (DPG) as our representation to model the structure of diagrams. We define syntactic parsing of diagrams as learning to infer DPGs for diagrams and study semantic interpretation and reasoning of diagrams in the context of diagram question answering. We devise an LSTM-based method for syntactic parsing of diagrams and introduce a DPG-based attention model for diagram question answering. We compile a new dataset of diagrams with exhaustive annotations of constituents and relationships for about 5,000 diagrams and 15,000 questions and answers. Our results show the significance of our models for syntactic parsing and question answering in diagrams using DPGs. <s> BIB005
The very early computer program, BEATRIX  , BIB001 , parses the English text and diagram components of the elementary physics problems together by establishing the coreference between the text and diagram. Watanabe et al. proposed a framework to combine layout information and natural language to analyze the pictorial book of flora diagrams BIB002 . An overview of the research on integration of visual and linguistic information was provided in the survey paper by Srihar BIB003 . However, these early approaches rely on written rules or manual regulations, i.e., the visual elements needed to be recognized with human intervention and their performances were usually dependent on specified diagrams. G-ALINGER BIB004 is an algorithmic work that addresses the geometry understanding and text understanding simultaneously. To detect primitives from a geometric diagram, Hough transform  is first applied to initialize lines and circles segments. An objective function that incorporates pixel coverage, visual coherence and textual-visual alignment. The function is submodular and a greedy algorithm is designed to pick the primitive with the maximum gain in each iteration. The algorithm stops when no positive gain can be obtained according to the objective function. In BIB005 , the problem of visual understanding is addressed in the context of science diagrams. The objective is to identify the graphic representation for the visual entities and their relations such as temporal transitions, phase transformations and inter object dependencies. An LSTM-based network is proposed for syntactic parsing of diagrams and learns the graphic structure.