Visualizing Natural Language Descriptions: A Survey <s> WordsEye <s> Natural language is an easy and effective medium for describing visual ideas and mental images. Thus, we foresee the emergence of language-based 3D scene generation systems to let ordinary users quickly create 3D scenes without having to learn special software, acquire artistic skills, or even touch a desktop window-oriented interface. WordsEye is such a system for automatically converting text into representative 3D scenes. WordsEye relies on a large database of 3D models and poses to depict entities and actions. Every 3D model can have associated shape displacements, spatial tags, and functional properties to be used in the depiction process. We describe the linguistic analysis and depiction techniques used by WordsEye along with some general strategies by which more abstract concepts are made depictable. <s> BIB001 </s> Visualizing Natural Language Descriptions: A Survey <s> WordsEye <s> MICA is a dependency parser which returns deep dependency representations, is fast, has state-of-the-art performance, and is freely available. <s> BIB002 </s> Visualizing Natural Language Descriptions: A Survey <s> WordsEye <s> 3D graphics scenes are difficult to create, requiring users to learn and utilize a series of complex menus, dialog boxes, and often tedious direct manipulation techniques. By giving up some amount of control afforded by such interfaces we have found that users can use natural language to quickly and easily create a wide variety of 3D scenes. Natural language offers an interface that is intuitive and immediately accessible by anyone, without requiring any special skill or training. The WordsEye system (http://www.wordseye.com) has been used by several thousand users on the web to create over 10,000 scenes. The system relies on a large database of 3D models and poses to depict entities and actions. We describe how the current version of the system incorporates the type of lexical and real-world knowledge needed to depict scenes from language. <s> BIB003 </s> Visualizing Natural Language Descriptions: A Survey <s> WordsEye <s> Spatial relations play an important role in our understanding of language. In particular, they are a crucial component in descriptions of scenes in the world. WordsEye (www.wordseye.com) is a system for automatically converting natural language text into 3D scenes representing the meaning of that text. Natural language offers an interface to scene generation that is intuitive and immediately approachable by anyone, without any special skill or training. WordsEye has been used by several thousand users on the web to create approximately 15,000 fully rendered scenes. We describe how the system incorporates geometric and semantic knowledge about objects and their parts and the spatial relations that hold among these in order to depict spatial relations in 3D scenes. <s> BIB004 </s> Visualizing Natural Language Descriptions: A Survey <s> WordsEye <s> WordsEye is a system for converting from English text into three-dimensional graphical scenes that represent that text. It works by performing syntactic and semantic analyses on the input text, producing a description of the arrangement of objects in a scene. At the core of WordsEye is the Scenario-Based Lexical Knowledge Resource (SBLR), a unified knowledge base and representational system for expressing lexical and real-world knowledge needed to depict scenes from text. This paper explores information collection methods for building the SBLR, using Amazon's Mechanical Turk (AMT) and manual normalization of raw AMT data. The paper follows with manual review of existing relations in the SBLR and classification of the AMT data into existing and new semantic relations. Since manual annotation is a time-consuming and expensive approach, we also explored the use of automatic normalization of AMT data through log-odds and log-likelihood ratios extracted from the English Gigaword corpus, as well as through WordNet similarity measures. <s> BIB005 </s> Visualizing Natural Language Descriptions: A Survey <s> WordsEye <s> This paper introduces Vignette Semantics, a lexical semantic theory based on Frame Semantics that represents conceptual and graphical relations. We also describe a lexical resource that implements this theory, VigNet, and its application in text-to-scene generation. <s> BIB006 </s> Visualizing Natural Language Descriptions: A Survey <s> WordsEye <s> We investigate using Amazon Mechanical Turk (AMT) for building a low-level description corpus and populating VigNet, a comprehensive semantic resource that we will use in a text-to-scene generation system. To depict a picture of a location, VigNet should contain the knowledge about the typical objects in that location and the arrangements of those objects. Such information is mostly common-sense knowledge that is taken for granted by human beings and is not stated in existing lexical resources and in text corpora. In this paper we focus on collecting objects of locations using AMT. Our results show that it is a promising approach. <s> BIB007 </s> Visualizing Natural Language Descriptions: A Survey <s> WordsEye <s> WordsEye is a system for automatically converting a text description of a scene into a 3D image. In converting a text description into a corresponding 3D scene, it is necessary to map objects and locations specified in the text into the actual 3D objects. Individual objects typically correspond to single 3D models, but locations (e.g. a living room) are typically an ensemble of objects. Prototypical mappings from locations to objects and their relations are called location vignettes, which are not present in existing lexical resources. In this paper we propose a new methodology using Amazon's Mechanical Turk to collect semantic information for location vignettes. Our preliminary results show that this is a promising approach. <s> BIB008 </s> Visualizing Natural Language Descriptions: A Survey <s> WordsEye <s> Text-to-scene conversion requires knowledge about how actions and locations are expressed in language and realized in the world. To provide this knowlege, we are creating a lexical resource (VigNet) that extends FrameNet by creating a set of intermediate frames (vignettes) that bridge between the high-level semantics of FrameNet frames and a new set of low-level primitive graphical frames. Vignettes can be thought of as a link between function and form ‐ between what a scene means and what it looks like. In this paper, we describe the set of primitive graphical frames and the functional properties of 3D objects (affordances) we use in this decomposition. We examine the methods and tools we have developed to populate VigNet with a large number of action and location vignettes. <s> BIB009
WordsEye BIB001 is designed to generate 3D scenes containing environment, objects, characters, attributes, poses, kinematics, and spatial relations. The system input is a set of textual descriptions that can include information about actions, spatial relations, and object attributes. The system consists of two main components, including a linguistic analyzer and a scene depicter. The linguistic analyzer is equipped with a POS tagger and a statistical parser. In the early version, an analyzer was implemented in common Lisp and, later, MICA parser BIB002 was exploited as well. It parses the input text and constructs a dependency structure that represents the dependencies among the words to facilitate the semantic analysis. This structure is then utilized to construct a semantic representation in which objects, actions, and relations are represented in terms of semantic frames BIB003 . The words with noun POS tags are associated with 3D objects and their associated hyponyms (i.e., words with an is-a semantic relation) and hypernyms (i.e., words with inverse semantic relation of hyponymy) are acquired using WordNet. The spatial relations are captured using a set of predefined spatial patterns based on the dependency structure. The words with a verb POS tag are associated with a set of parametrized functions that indicate the effects of the verbs. In a recent development, lexical knowledge extracted from WordNet and FrameNet are semi-manually refined to construct a Scenario-Based Lexical Knowledge Resource (SBLR), which is essentially a lexical knowledgebase tailored to represent the lexical and common-sense knowledge for text-to-scene conversion purposes BIB004 . The knowledge in SBLR is represented by VigNet BIB006 BIB009 which is an extension of FrameNet and consists of a set of intermediate frames called Vignettes that bridge the semantic gap between the semantic frames of FrameNet and the low-level graphical frames. VigNet also contains implicit knowledge of a restricted set of environments such as a kitchen. This knowledge is a remedy for missing common-sense facts within the natural language descriptions and is acquired through manual descriptions of the pictures gathered from Amazon Mechanical Turk (AMT)  -an online crowd-sourcing framework for data collection using Human Intelligence Task (HITs). The collected corpus is processed using naive text processing techniques to populate the VigNet with extracted Vignettes BIB007 BIB005 BIB008 . The Depiction module converts a set of semantic frames into a set of low-level graphical specifications. For this purpose, it uses a set of depiction rules to convert the objects, actions, relations, and attributes from the extracted semantic representation to their realizable visual counterpart. The geometric information of the objects is manually tagged and attached to the 3D models. This component also employs a set of transduction rules to solve the implicit and conflicting constraints while positioning the objects in the scene in an incremental manner. As soon as the layout is completed, the static scene is rendered using OpenGL. WordsEye relies on its huge off-line rule-base and data repositories. Its semantic database consists of 15,000 nouns and 2,300 verbs, whereas its visual database consists of 2,200 3D models and 10,000 images. Different features of these models, such as geometric shape, type, flexibility, embeddability, and so on, are manually annotated. As an instance, all objects with a long thin vertical base are annotated as stem. WordsEye also contains a large set of rules including spatial rules, depiction rules, and transduction rules. For example, it contains three rules for kicking action whose firing strengths are evaluated based on the type of object to be kicked. WordsEye has been utilized by a few thousand on-line users to create 15,000 static scenes. Although it has achieved a good degree of success, the allowed input language for describing the scenes is stilted BIB003 . Another problem is that WordsEye is not interactive and does not exploit the user's feedbacks. Also, there is no interface provided for adding new knowledge or rules to the system and one would require hard-wiring them into the system.