Visualizing Natural Language Descriptions: A Survey <s> Systems That Learn Visual Clues <s> This paper details a new approach for learning a discriminative model of object classes, incorporating texture, layout, and context information efficiently. The learned model is used for automatic visual understanding and semantic segmentation of photographs. Our discriminative model exploits texture-layout filters, novel features based on textons, which jointly model patterns of texture and their spatial layout. Unary classification and feature selection is achieved using shared boosting to give an efficient classifier which can be applied to a large number of classes. Accurate image segmentation is achieved by incorporating the unary classifier in a conditional random field, which (i) captures the spatial interactions between class labels of neighboring pixels, and (ii) improves the segmentation of specific object instances. Efficient training of the model on large datasets is achieved by exploiting both random feature selection and piecewise training methods. ::: ::: High classification and segmentation accuracy is demonstrated on four varied databases: (i) the MSRC 21-class database containing photographs of real objects viewed under general lighting conditions, poses and viewpoints, (ii) the 7-class Corel subset and (iii) the 7-class Sowerby database used in He et al. (Proceeding of IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 695---702, June 2004), and (iv) a set of video sequences of television shows. The proposed algorithm gives competitive and visually pleasing results for objects that are highly textured (grass, trees, etc.), highly structured (cars, faces, bicycles, airplanes, etc.), and even articulated (body, cow, etc.). <s> BIB001 </s> Visualizing Natural Language Descriptions: A Survey <s> Systems That Learn Visual Clues <s> We describe MSR SPLAT, a toolkit for language analysis that allows easy access to the linguistic analysis tools produced by the NLP group at Microsoft Research. The tools include both traditional linguistic analysis tools such as part-of-speech taggers, constituency and dependency parsers, and more recent developments such as sentiment detection and linguistically valid morphology. As we expand the tools we develop for our own research, the set of tools available in MSR SPLAT will be extended. The toolkit is accessible as a web service, which can be used from a broad set of programming languages. <s> BIB002 </s> Visualizing Natural Language Descriptions: A Survey <s> Systems That Learn Visual Clues <s> We present AttribIt, an approach for people to create visual content using relative semantic attributes expressed in linguistic terms. During an off-line processing step, AttribIt learns semantic attributes for design components that reflect the high-level intent people may have for creating content in a domain (e.g. adjectives such as "dangerous", "scary" or "strong") and ranks them according to the strength of each learned attribute. Then, during an interactive design session, a person can explore different combinations of visual components using commands based on relative attributes (e.g. "make this part more dangerous"). Novel designs are assembled in real-time as the strengths of selected attributes are varied, enabling rapid, in-situ exploration of candidate designs. We applied this approach to 3D modeling and web design. Experiments suggest this interface is an effective alternative for novices performing tasks with high-level design goals. <s> BIB003 </s> Visualizing Natural Language Descriptions: A Survey <s> Systems That Learn Visual Clues <s> Sentences that describe visual scenes contain a wide variety of information pertaining to the presence of objects, their attributes and their spatial relations. In this paper we learn the visual features that correspond to semantic phrases derived from sentences. Specifically, we extract predicate tuples that contain two nouns and a relation. The relation may take several forms, such as a verb, preposition, adjective or their combination. We model a scene using a Conditional Random Field (CRF) formulation where each node corresponds to an object, and the edges to their relations. We determine the potentials of the CRF using the tuples extracted from the sentences. We generate novel scenes depicting the sentences' visual meaning by sampling from the CRF. The CRF is also used to score a set of scenes for a text-based image retrieval task. Our results show we can generate (retrieve) scenes that convey the desired semantic meaning, even when scenes (queries) are described by multiple sentences. Significant improvement is found over several baseline approaches. <s> BIB004
The systems mentioned so far only utilize textual clues either as a set of pre-defined rules or a set of models learned from corpora. A new trend in text-to-scene conversion systems is learning the associations between both textual and visual clues. These research works follow the approach used by text-to-picture conversion systems by focusing on learning the visual features from available image database and extracting associations between visual and textual features to automate the visualization process.However, contrary to the text-to-picture conversion systems, they use associations to position the objects within a static scene rather than selecting representative pictures. AttribIt BIB003 is designed to help the users create visual content using subjective attributes such as dangerous airplane. This system provides the user with a set of 3D parts of a model of interest and helps her with assembling those components to construct a plausible model. It exploits AMT crowd-sourcing and presents the volunteers with a set of 3D models of different parts of objects such as airplane wings and asks them to compare each and every pair of models using adjectives. It then ranks the associations between the components and gathered attributes using the Support Vector Machine (SVM) classifier. The learned model is used along with a GUI to directly capture the semantic attributes and to provide the user with corresponding parts of the model. A promising data-driven system developed at Microsoft Research Center is introduced in BIB004 . This system learns the visual features from abstract scenes, extracts the semantic information from corresponding corpus, and learns the associations between extracted visual features and semantics to generate new scenes based on a set of unseen natural language descriptions. Similar to AttribIt, this system utilizes AMT for gathering a training dataset. It exploits Conditional Random Field (CRF) BIB001 to extract objects and their occurrences, attributes, and positions. After extracting visual features, the system extracts semantics in the form of predicate tuples using semantic role analysis BIB002 . For scene generation, the system learns the associations between the predicate tuples and the visual features based on their co-occurrences using highest mutual information and then uses these associations to generate a new scene. This system is purely data driven and learns how to generate static 2D scenes by observing available scenes and corresponding descriptors. However, it does not support online learning. The authors have only used their system on a simple 2D scenario of children's playground. Therefore, it is not clear whether their approach can generate satisfactory scenes in 3D scenarios as well. The system also lacks a strong semantic analysis for capturing more general dependencies.