Visualizing Natural Language Descriptions: A Survey <s> System Developed at Stanford University <s> We address the grounding of natural language to concrete spatial constraints, and inference of implicit pragmatics in 3D environments. We apply our approach to the task of text-to-3D scene generation. We present a representation for common sense spatial knowledge and an approach to extract it from 3D scene data. In text-to3D scene generation, a user provides as input natural language text from which we extract explicit constraints on the objects that should appear in the scene. The main innovation of this work is to show how to augment these explicit constraints with learned spatial knowledge to infer missing objects and likely layouts for the objects in the scene. We demonstrate that spatial knowledge is useful for interpreting natural language and show examples of learned knowledge and generated 3D scenes. <s> BIB001 </s> Visualizing Natural Language Descriptions: A Survey <s> System Developed at Stanford University <s> We propose text-to-scene generation as an application for semantic parsing. This is an application that grounds semantics in a virtual world that requires understanding of common, everyday language. In text to scene generation, the user provides a textual description and the system generates a 3D scene. For example, Figure 1 shows the generated scene for the input text “there is a room with a chair and a computer”. This is a challenging, open-ended problem that prior work has only addressed in a limited way. Most of the technical challenges in text to scene generation stem from the difficulty of mapping language to formal representations of visual scenes, as well as an overall absence of real world spatial knowledge from current NLP systems. These issues are partly due to the omission in natural language of many facts about the world. When people describe scenes in text, they typically specify only important, relevant information. Many common sense facts are unstated (e.g., chairs and desks are typically on the floor). Therefore, we focus on inferring implicit relations that are likely to hold even if they are not explicitly stated by the input text. Text to scene generation offers a rich, interactive environment for grounded language that is familiar to everyone. The entities are common, everyday objects, and the knowledge necessary to address this problem is of general use across many domains. We present a system that leverages user interactionwith 3D scenes to generate training data for semantic parsing approaches. Previous semantic parsing work has dealt with grounding text to physical attributes and relations (Matuszek et al., 2012; Krishnamurthy and Kollar, 2013), generating text for referring to objects (FitzGerald et al., 2013) and with connecting language to spatial relationships (Golland et al., 2010; Artzi and Zettlemoyer, 2013). Semantic parsing methods can also be applied to many aspects of text to scene generation. Furthermore, work on parsing instructions to robots (Matuszek et al., 2013; Tellex et al., 2014) has analogues in the context of discourse about physical scenes. In this extended abstract, we formalize the text to scene generation problem and describe it as a task for semantic parsing methods. To motivate this problem, we present a prototype system that incorporates simple spatial knowledge, and parses natural text to a semantic representation. By learning priors on spatial knowledge (e.g., typical positions of objects, and common spatial relations) our system addresses inference of implicit spatial constraints. The user can interactively manipulate the generated scene with textual commands, enabling us to refine and expand learned priors. Our current system uses deterministic rules to map text to a scene representation but we plan to explore training a semantic parser from data. We can leverage our system to collect user interactions for training data. Crowdsourcing is a promising avenue for obtaining a large scale dataset. <s> BIB002 </s> Visualizing Natural Language Descriptions: A Survey <s> System Developed at Stanford University <s> We describe the design and use of the Stanford CoreNLP toolkit, an extensible pipeline that provides core natural language analysis. This toolkit is quite widely used, both in the research NLP community and also among commercial and government users of open source NLP technology. We suggest that this follows from a simple, approachable design, straightforward interfaces, the inclusion of robust and good quality analysis components, and not requiring use of a large amount of associated baggage. <s> BIB003
Contrary to previous systems, the system developed at Stanford University  BIB001 BIB002 infers the implicit relations and partially supports interactive scene manipulation and active learning. In this system, a scene template is generated from an input text and converted to a geometric graph that is then utilized to render a static scene. The scene template constructed from the input text using Stanford CoreNLP language processing tool BIB003 is a graph with objects as its vertices and relations as its edges. The objects are recognized by detecting nouns that are considered as visualizable according to WordNet. The words with an adjective tag within the noun phrases are extracted to identify the attributes of the objects. The spatial relations are extracted using a set of pre-defined patterns. Natural language descriptions usually do not contain common-sense facts about the spatial arrangements. To alleviate this challenge, the system uses conditional probability to model the object occurrences and hierarchy priors and exploits Bayes's rule to infer the implicit spatial arrangements. The inferred knowledge is then inserted into the scene template graph. For example, for a sample input text, put the cake on the table it can infer to put the cake on a plate and put the plate on the table. The geometric graph contains a set of 3D model instances that correspond to the objects within the scene template and their associated spatial arrangements. This graph is used directly to render the static 3D scene. The system also supports interactive scene manipulation and active learning. The user can add new objects to defined positions and remove the existing objects. She can also select an object within the scene and annotate it. The system modifies its probabilistic model of support hierarchy by observing how users design the scenes. For example, if a user asks the system to put a cup on the table, the system increases the probability of co-occurrence of the cup and the table and the probability of table supporting a cup. This system surpasses previous text-to-scene conversion systems in terms of adaptive behavior and interactivity. However, in terms of language understanding and richness of the model repository, WordsEye and AVDT outperform this system.