We present a neural network architecture to predict a point in color space from the sequence of characters in the color's name. Using large scale color--name pairs obtained from an online color design forum, we evaluate our model on a "color Turing test" and find that, given a name, the colors predicted by our model are preferred by annotators to color names created by humans. Our datasets and demo system are available online at this http URL.
Closer to NLP, the relationship between visual stimuli and their linguistic descriptions by humans has been explored extensively through automatic text generation from images @cite_23 @cite_14 @cite_30 . Color association with word semantics has also been investigated in several previous papers @cite_22 @cite_15 @cite_10 @cite_19 .
@cite_30 @cite_14 @cite_22 @cite_19 @cite_23 @cite_15 @cite_10
2043464897 2173294139 2226480163 2110933980
Abstract In this study, we investigated the spatio-temporal patterns of category-specific cortical activation elicited by the visual presentation of words whose meaning relates to a color or to a visual form or shape. We recorded the event-related potentials (ERPs) of ten healthy, right-handed volunteers while they passively read words presented tachistoscopically. As early as 150 ms after stimulus onset, the ERPs revealed significant neurophysiological differences between words and strings of hash marks. Around 200 ms after stimulus onset, we found significant differences in the ERPs elicited by color- and form-related words. We used minimum norm current estimates to investigate the spatial location of these differences. This revealed that, at the 150 ms peak, the activation advantage of words over sequences of hash marks was located in a left posterior area, proximal to what has previously been called the visual word form area. At the 200 ms peak, the advantage of words related to colors over words related to forms seemed to reside in temporal cortical areas, whereas the form-related words elicited greater activation in frontal areas than color-related words. These results provide evidence for early access to detailed category-specific representations of word meaning, with subtle differences in meaning being reflected in the activation of different cortical areas, as early as 200 ms after stimulus presentation. In line with previous studies, these differences can be related to the areas involved in the conceptual processing of sensory (visual) and action-related information. We propose a model to learn visually grounded word embeddings (vis-w2v) to capture visual notions of semantic relatedness. While word embeddings trained using text have been extremely successful, they cannot uncover notions of semantic relatedness implicit in our visual world. For instance, although "eats" and "stares at" seem unrelated in text, they share semantics visually. When people are eating something, they also tend to stare at the food. Grounding diverse relations like "eats" and "stares at" into vision remains challenging, despite recent progress in vision. We note that the visual grounding of words depends on semantics, and not the literal pixels. We thus use abstract scenes created from clipart to provide the visual grounding. We find that the embeddings we learn capture fine-grained, visually grounded notions of semantic relatedness. We show improvements over text-only word embeddings (word2vec) on three tasks: common-sense assertion classification, visual paraphrasing and text-based image retrieval. Our code and datasets are available online. In this paper we propose the construction of linguistic descriptions of images. This is achieved through the extraction of scene description graphs (SDGs) from visual scenes using an automatically constructed knowledge base. SDGs are constructed using both vision and reasoning. Specifically, commonsense reasoning is applied on (a) detections obtained from existing perception methods on given images, (b) a "commonsense" knowledge base constructed using natural language processing of image annotations and (c) lexical ontological knowledge from resources such as WordNet. Amazon Mechanical Turk(AMT)-based evaluations on Flickr8k, Flickr30k and MS-COCO datasets show that in most cases, sentences auto-constructed from SDGs obtained by our method give a more relevant and thorough description of an image than a recent state-of-the-art image caption based approach. Our Image-Sentence Alignment Evaluation results are also comparable to that of the recent state-of-the art approaches. Humans use rich natural language to describe and communicate visual perceptions. In order to provide natural language descriptions for visual content, this paper combines two important ingredients. First, we generate a rich semantic representation of the visual content including e.g. object and activity labels. To predict the semantic representation we learn a CRF to model the relationships between different components of the visual input. And second, we propose to formulate the generation of natural language as a machine translation problem using the semantic representation as source language and the generated sentences as target language. For this we exploit the power of a parallel corpus of videos and textual descriptions and adapt statistical machine translation to translate between our two languages. We evaluate our video descriptions on the TACoS dataset, which contains video snippets aligned with sentence descriptions. Using automatic evaluation and human judgments we show significant improvements over several baseline approaches, motivated by prior work. Our translation approach also shows improvements over related work on an image description task.