Visualizing Natural Language Descriptions: A Survey <s> Text-to-Picture <s> In this paper, we present an approach towards automated story picturing based on mutual reinforcement principle. Story picturing refers to the process of illustrating a story with suitable pictures. In our approach, semantic keywords are extracted from the story text and an annotated image database is searched to form an initial picture pool. Thereafter, a novel image ranking scheme automatically determines the importance of each image. Both lexical annotations and visual content of an image play a role in determining its rank. Annotations are processed using the Wordnet to derive a lexical signature for each image. An integrated region based similarity is also calculated between each pair of images. An overall similarity measure is formed using lexical and visual features. In the end, a mutual reinforcement based rank is calculated for each image using the image similarity matrix. We also present a human behavior model based on a discrete state Markov process which captures the intuition for our technique. Experimental results have demonstrated the effectiveness of our scheme <s> BIB001 </s> Visualizing Natural Language Descriptions: A Survey <s> Text-to-Picture <s> We present an unsupervised approach to automated story picturing. Semantic keywords are extracted from the story, an annotated image database is searched. Thereafter, a novel image ranking scheme automatically determines the importance of each image. Both lexical annotations and visual content play a role in determining the ranks. Annotations are processed using the Wordnet. A mutual reinforcement-based rank is calculated for each image. We have implemented the methods in our Story Picturing Engine (SPE) system. Experiments on large-scale image databases are reported. A user study has been performed and statistical analysis of the results has been presented. <s> BIB002 </s> Visualizing Natural Language Descriptions: A Survey <s> Text-to-Picture <s> The bag-of-words model is one of the most popular representation methods for object categorization. The key idea is to quantize each extracted key point into one of visual words, and then represent each image by a histogram of the visual words. For this purpose, a clustering algorithm (e.g., K-means), is generally used for generating the visual words. Although a number of studies have shown encouraging results of the bag-of-words representation for object categorization, theoretical studies on properties of the bag-of-words model is almost untouched, possibly due to the difficulty introduced by using a heuristic clustering process. In this paper, we present a statistical framework which generalizes the bag-of-words representation. In this framework, the visual words are generated by a statistical process rather than using a clustering algorithm, while the empirical performance is competitive to clustering-based method. A theoretical analysis based on statistical consistency is presented for the proposed framework. Moreover, based on the framework we developed two algorithms which do not rely on clustering, while achieving competitive performance in object categorization when compared to clustering-based bag-of-words representations. <s> BIB003 </s> Visualizing Natural Language Descriptions: A Survey <s> Text-to-Picture <s> Textbooks have a direct bearing on the quality of education imparted to the students. Therefore, it is of paramount importance that the educational content of textbooks should provide rich learning experience to the students. Recent studies on understanding learning behavior suggest that the incorporation of digital visual material can greatly enhance learning. However, textbooks used in many developing regions are largely text-oriented and lack good visual material. We propose techniques for finding images from the web that are most relevant for augmenting a section of the textbook, while respecting the constraint that the same image is not repeated in different sections of the same chapter. We devise a rigorous formulation of the image assignment problem and present a polynomial time algorithm for solving the problem optimally. We also present two image mining algorithms that utilize orthogonal signals and hence obtain different sets of relevant images. Finally, we provide an ensembling algorithm for combining the assignments. To empirically evaluate our techniques, we use a corpus of high school textbooks in use in India. Our user study utilizing the Amazon Mechanical Turk platform indicates that the proposed techniques are able to obtain images that can help increase the understanding of the textbook material. <s> BIB004 </s> Visualizing Natural Language Descriptions: A Survey <s> Text-to-Picture <s> Automatic image annotation can be used to facilitate semantic search in large image databases. However, retrieval performance of the existing annotation schemes is far from the users' expectation. In this paper, we propose a novel method to automatically annotate image through the rules generated by support vector machines and decision trees. In order to obtain the rules, we collect a set of training regions by image segmentation, feature extraction and discretization. We first employ a support vector machine as a preprocessing technique to refine the input training data and then use it to improve the rules generated by decision tree learning. The preprocessing can effectively deal with the similar regions in an image as well. Moreover, we integrate the original rules to the modified ones, so as to formulate the complete and effective annotation rules. We can translate an unknown image into text by this algorithm, and the proposed system can retrieve images queried by both images and keywords. Experiments are carried out in a standard Corel dataset and images collected from the Web to test the accuracy and robustness of the proposed system. Experimental results show the proposed algorithm can annotate and retrieve images more efficiently than traditional learning algorithms. <s> BIB005 </s> Visualizing Natural Language Descriptions: A Survey <s> Text-to-Picture <s> The number of images on the World Wide Web has been increasing tremendously. Providing search services for images on the web has been an active research area. Web images are often surrounded by different associated texts like ALT text, surrounding text, image filename, html page title etc. Many popular internet search engines make use of these associated texts while indexing images and give higher importance to the terms present in ALT text. But, a recent study has shown that around half of the images on the web have no ALT text. So, predicting the ALT text of an image in a web page would be of great use in web image retrieval. We propose an approach on top of term co-occurrence approach proposed in the literature to ALT text prediction. Our results show that our approach and the simple term co-occurrence approach produce almost the same results. We analyze both the methods and describe the usage of the methods in different situations. We build an image annotation system on top of our proposed approach and compare the results with the image annotation system built on top of the term co-occurrence approach. Preliminary experiments on a set of 1000 images show that our proposed approach performs well over the simple term co-occurrence approach for web image annotation. <s> BIB006 </s> Visualizing Natural Language Descriptions: A Survey <s> Text-to-Picture <s> This paper presents a semi-automatic tool, called IGAnn (Interactive image ANNotation), that assists users in annotating textual labels with images. IGAnn performs an interactive retrieval-like procedure: the system presents the user with images that have higher confidences, and then the user determines which images are actually relevant or irrelevant for a specified label. By collecting relevant and irrelevant images of iterations, a hierarchical classifier associated with the specified label is built using our proposed semi-supervised approach to compute confidence values of unlabeled images. This paper describes the system interface of IGAnn and also demonstrates quantitative experiments of our proposed approach. <s> BIB007
Text-to-picture conversion is probably the simplest method for visualizing natural language descriptions BIB004 BIB001 BIB002  . It treats the problem of mapping natural language descriptions to a visual representation as a data-driven image retrieval and ranking problem and tries to solve it using the foundations of commercial Web-based image search engines. In this approach, descriptive terms or constituents that represent the main concepts of the text are extracted using text-mining techniques such as bag-ofwords, named entities, and N-gram models. It is assumed that an annotated dataset of images is available. In case of automatic annotation, it is common to collect a repository of multimodal information containing both images and text and then to use the co-occurring text around images to annotate them. In Web-based image retrieval systems, this process is carried out by exploiting the surrounding text of the images and the text appearing within HTML tags. The extracted text is then tokenized and a subset of terms is selected and scored to determine the weighted annotations of the corresponding images BIB005 BIB007   BIB006 . The extracted concepts are then matched against the image annotations and a subset of images are retrieved and ranked for a given concept based on some predefined similarity measures. Finally, for each concept, the retrieved images with the highest rank are illustrated in the same order that their corresponding concepts appear in the text. This approach inherits the solid theoretical foundations of search engines. Also, because of exploiting statistical information retrieval rather than natural language understanding, the text-to-picture conversion approach is computationally efficient BIB003 . However, it does not result in expected visualization due to three main reasons: (1) It cannot capture the deep semantics embedded within the natural language descriptions, (2) the visualization is restricted to available images, and (3) it cannot interpolate the inbetween visual information. This approach is not the main focus of this survey and hence only two systems are discussed.