A Comprehensive Survey of Deep Learning for Image Captioning <s> Semantic-Concept-Based Image <s> Humans can prepare concise descriptions of pictures, focusing on what they find important. We demonstrate that automatic methods can do so too. We describe a system that can compute a score linking an image to a sentence. This score can be used to attach a descriptive sentence to a given image, or to obtain images that illustrate a given sentence. The score is obtained by comparing an estimate of meaning obtained from the image to one obtained from the sentence. Each estimate of meaning comes from a discriminative procedure that is learned us-ingdata. We evaluate on a novel dataset consisting of human-annotated images. While our underlying estimate of meaning is impoverished, it is sufficient to produce very good quantitative results, evaluated with a novel score that can account for synecdoche. <s> BIB001 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Semantic-Concept-Based Image <s> Studying natural language, and especially how people describe the world around them can help us better understand the visual world. In turn, it can also help us in the quest to generate natural language that describes this world in a human manner. We present a simple yet effective approach to automatically compose image descriptions given computer vision based inputs and using web-scale n-grams. Unlike most previous work that summarizes or retrieves pre-existing text relevant to an image, our method composes sentences entirely from scratch. Experimental results indicate that it is viable to generate simple textual descriptions that are pertinent to the specific content of an image, while permitting creativity in the description -- making for more human-like annotations than previous approaches. <s> BIB002 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Semantic-Concept-Based Image <s> We present a holistic data-driven approach to image description generation, exploiting the vast amount of (noisy) parallel image data and associated natural language descriptions available on the web. More specifically, given a query image, we retrieve existing human-composed phrases used to describe visually similar images, then selectively combine those phrases to generate a novel description for the query image. We cast the generation process as constraint optimization problems, collectively incorporating multiple interconnected aspects of language composition for content planning, surface realization and discourse structure. Evaluation by human annotators indicates that our final system generates more semantically correct and linguistically appealing descriptions than two nontrivial baselines. <s> BIB003 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Semantic-Concept-Based Image <s> Describing the main event of an image involves identifying the objects depicted and predicting the relationships between them. Previous approaches have represented images as unstructured bags of regions, which makes it difficult to accurately predict meaningful relationships between regions. In this paper, we introduce visual dependency representations to capture the relationships between the objects in an image, and hypothesize that this representation can improve image description. We test this hypothesis using a new data set of region-annotated images, associated with visual dependency representations and gold-standard descriptions. We describe two template-based description generation models that operate over visual dependency representations. In an image description task, we find that these models outperform approaches that rely on object proximity or corpus information to generate descriptions on both automatic measures and on human judgements. <s> BIB004 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Semantic-Concept-Based Image <s> We introduce a model for bidirectional retrieval of images and sentences through a deep, multi-modal embedding of visual and natural language data. Unlike previous models that directly map images or sentences into a common embedding space, our model works on a finer level and embeds fragments of images (objects) and fragments of sentences (typed dependency tree relations) into a common space. We then introduce a structured max-margin objective that allows our model to explicitly associate these fragments across modalities. Extensive experimental evaluation shows that reasoning on both the global level of images and sentences and the finer level of their respective fragments improves performance on image-sentence retrieval tasks. Additionally, our model provides interpretable predictions for the image-sentence retrieval task since the inferred inter-modal alignment of fragments is explicit. <s> BIB005 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Semantic-Concept-Based Image <s> In this paper, we present a multimodal Recurrent Neural Network (m-RNN) model for generating novel image captions. It directly models the probability distribution of generating a word given previous words and an image. Image captions are generated by sampling from this distribution. The model consists of two sub-networks: a deep recurrent neural network for sentences and a deep convolutional network for images. These two sub-networks interact with each other in a multimodal layer to form the whole m-RNN model. The effectiveness of our model is validated on four benchmark datasets (IAPR TC-12, Flickr 8K, Flickr 30K and MS COCO). Our model outperforms the state-of-the-art methods. In addition, we apply the m-RNN model to retrieval tasks for retrieving images or sentences, and achieves significant performance improvement over the state-of-the-art methods which directly optimize the ranking objective function for retrieval. The project page of this work is: www.stat.ucla.edu/~junhua.mao/m-RNN.html . <s> BIB006 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Semantic-Concept-Based Image <s> Generating a novel textual description of an image is an interesting problem that connects computer vision and natural language processing. In this paper, we present a simple model that is able to generate descriptive sentences given a sample image. This model has a strong focus on the syntax of the descriptions. We train a purely bilinear model that learns a metric between an image representation (generated from a previously trained Convolutional Neural Network) and phrases that are used to described them. The system is then able to infer phrases from a given image sample. Based on caption syntax statistics, we propose a simple language model that can produce relevant descriptions for a given test image using the phrases inferred. Our approach, which is considerably simpler than state-of-the-art models, achieves comparable results on the recently release Microsoft COCO dataset. <s> BIB007 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Semantic-Concept-Based Image <s> We present a model that generates natural language descriptions of images and their regions. Our approach leverages datasets of images and their sentence descriptions to learn about the inter-modal correspondences between language and visual data. Our alignment model is based on a novel combination of Convolutional Neural Networks over image regions, bidirectional Recurrent Neural Networks over sentences, and a structured objective that aligns the two modalities through a multimodal embedding. We then describe a Multimodal Recurrent Neural Network architecture that uses the inferred alignments to learn to generate novel descriptions of image regions. We demonstrate that our alignment model produces state of the art results in retrieval experiments on Flickr8K, Flickr30K and MSCOCO datasets. We then show that the generated descriptions significantly outperform retrieval baselines on both full images and on a new dataset of region-level annotations. <s> BIB008 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Semantic-Concept-Based Image <s> In this paper we explore the bi-directional mapping between images and their sentence-based descriptions. Critical to our approach is a recurrent neural network that attempts to dynamically build a visual representation of the scene as a caption is being generated or read. The representation automatically learns to remember long-term visual concepts. Our model is capable of both generating novel captions given an image, and reconstructing visual features given an image description. We evaluate our approach on several tasks. These include sentence generation, sentence retrieval and image retrieval. State-of-the-art results are shown for the task of generating novel image descriptions. When compared to human generated captions, our automatically generated captions are equal to or preferred by humans 21.0% of the time. Results are better than or comparable to state-of-the-art results on the image and sentence retrieval tasks for methods using similar visual features. <s> BIB009 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Semantic-Concept-Based Image <s> Models based on deep convolutional networks have dominated recent image interpretation tasks; we investigate whether models which are also recurrent, or “temporally deep”, are effective for tasks involving sequences, visual and otherwise. We develop a novel recurrent convolutional architecture suitable for large-scale visual learning which is end-to-end trainable, and demonstrate the value of these models on benchmark video recognition tasks, image description and retrieval problems, and video narration challenges. In contrast to current models which assume a fixed spatio-temporal receptive field or simple temporal averaging for sequential processing, recurrent convolutional models are “doubly deep” in that they can be compositional in spatial and temporal “layers”. Such models may have advantages when target concepts are complex and/or training data are limited. Learning long-term dependencies is possible when nonlinearities are incorporated into the network state updates. Long-term RNN models are appealing in that they directly can map variable-length inputs (e.g., video frames) to variable length outputs (e.g., natural language text) and can model complex temporal dynamics; yet they can be optimized with backpropagation. Our recurrent long-term models are directly connected to modern visual convnet models and can be jointly trained to simultaneously learn temporal dynamics and convolutional perceptual representations. Our results show such models have distinct advantages over state-of-the-art models for recognition or generation which are separately defined and/or optimized. <s> BIB010 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Semantic-Concept-Based Image <s> In this paper, we address the task of learning novel visual concepts, and their interactions with other concepts, from a few images with sentence descriptions. Using linguistic context and visual features, our method is able to efficiently hypothesize the semantic meaning of new words and add them to its word dictionary so that they can be used to describe images which contain these novel concepts. Our method has an image captioning module based on [38] with several improvements. In particular, we propose a transposed weight sharing scheme, which not only improves performance on image captioning, but also makes the model more suitable for the novel concept learning task. We propose methods to prevent overfitting the new concepts. In addition, three novel concept datasets are constructed for this new task, and are publicly available on the project page. In the experiments, we show that our method effectively learns novel visual concepts from a few examples without disturbing the previously learned concepts. The project page is: www.stat.ucla.edu/junhua. mao/projects/child_learning.html. <s> BIB011 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Semantic-Concept-Based Image <s> Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing. In this paper, we present a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image. The model is trained to maximize the likelihood of the target description sentence given the training image. Experiments on several datasets show the accuracy of the model and the fluency of the language it learns solely from image descriptions. Our model is often quite accurate, which we verify both qualitatively and quantitatively. For instance, while the current state-of-the-art BLEU-1 score (the higher the better) on the Pascal dataset is 25, our approach yields 59, to be compared to human performance around 69. We also show BLEU-1 score improvements on Flickr30k, from 56 to 66, and on SBU, from 19 to 28. Lastly, on the newly released COCO dataset, we achieve a BLEU-4 of 27.7, which is the current state-of-the-art. <s> BIB012 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Semantic-Concept-Based Image <s> Inspired by recent work in machine translation and object detection, we introduce an attention based model that automatically learns to describe the content of images. We describe how we can train this model in a deterministic manner using standard backpropagation techniques and stochastically by maximizing a variational lower bound. We also show through visualization how the model is able to automatically learn to fix its gaze on salient objects while generating the corresponding words in the output sequence. We validate the use of attention with state-of-the-art performance on three benchmark datasets: Flickr9k, Flickr30k and MS COCO. <s> BIB013 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Semantic-Concept-Based Image <s> We propose a deep convolutional neural network architecture codenamed Inception that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14). The main hallmark of this architecture is the improved utilization of the computing resources inside the network. By a carefully crafted design, we increased the depth and width of the network while keeping the computational budget constant. To optimize quality, the architectural decisions were based on the Hebbian principle and the intuition of multi-scale processing. One particular incarnation used in our submission for ILSVRC14 is called GoogLeNet, a 22 layers deep network, the quality of which is assessed in the context of classification and detection. <s> BIB014 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Semantic-Concept-Based Image <s> Convolutional networks are powerful visual models that yield hierarchies of features. We show that convolutional networks by themselves, trained end-to-end, pixels-to-pixels, exceed the state-of-the-art in semantic segmentation. Our key insight is to build “fully convolutional” networks that take input of arbitrary size and produce correspondingly-sized output with efficient inference and learning. We define and detail the space of fully convolutional networks, explain their application to spatially dense prediction tasks, and draw connections to prior models. We adapt contemporary classification networks (AlexNet [20], the VGG net [31], and GoogLeNet [32]) into fully convolutional networks and transfer their learned representations by fine-tuning [3] to the segmentation task. We then define a skip architecture that combines semantic information from a deep, coarse layer with appearance information from a shallow, fine layer to produce accurate and detailed segmentations. Our fully convolutional network achieves state-of-the-art segmentation of PASCAL VOC (20% relative improvement to 62.2% mean IU on 2012), NYUDv2, and SIFT Flow, while inference takes less than one fifth of a second for a typical image. <s> BIB015 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Semantic-Concept-Based Image <s> Automatically generating a natural language description of an image has attracted interests recently both because of its importance in practical applications and because it connects two major artificial intelligence fields: computer vision and natural language processing. Existing approaches are either top-down, which start from a gist of an image and convert it into words, or bottom-up, which come up with words describing various aspects of an image and then combine them. In this paper, we propose a new algorithm that combines both approaches through a model of semantic attention. Our algorithm learns to selectively attend to semantic concept proposals and fuse them into hidden states and outputs of recurrent neural networks. The selection and fusion form a feedback connecting the top-down and bottom-up computation. We evaluate our algorithm on two public benchmarks: Microsoft COCO and Flickr30K. Experimental results show that our algorithm significantly outperforms the state-of-the-art approaches consistently across different evaluation metrics. <s> BIB016 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Semantic-Concept-Based Image <s> Attributes possess appealing properties and benefit many computer vision problems, such as object recognition, learning with humans in the loop, and image retrieval. Whereas the existing work mainly pursues utilizing attributes for various computer vision problems, we contend that the most basic problem—how to accurately and robustly detect attributes from images—has been left under explored. Especially, the existing work rarely explicitly tackles the need that attribute detectors should generalize well across different categories, including those previously unseen. Noting that this is analogous to the objective of multi-source domain generalization, if we treat each category as a domain, we provide a novel perspective to attribute detection and propose to gear the techniques in multi-source domain generalization for the purpose of learning cross-category generalizable attribute detectors. We validate our understanding and approach with extensive experiments on four challenging datasets and three different problems. <s> BIB017 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Semantic-Concept-Based Image <s> Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation. <s> BIB018 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Semantic-Concept-Based Image <s> Automatically describing an image with a natural language has been an emerging challenge in both fields of computer vision and natural language processing. In this paper, we present Long Short-Term Memory with Attributes (LSTM-A) - a novel architecture that integrates attributes into the successful Convolutional Neural Networks (CNNs) plus Recurrent Neural Networks (RNNs) image captioning framework, by training them in an end-to-end manner. To incorporate attributes, we construct variants of architectures by feeding image representations and attributes into RNNs in different ways to explore the mutual but also fuzzy relationship between them. Extensive experiments are conducted on COCO image captioning dataset and our framework achieves superior results when compared to state-of-the-art deep models. Most remarkably, we obtain METEOR/CIDEr-D of 25.2%/98.6% on testing data of widely used and publicly available splits in (Karpathy&Fei-Fei, 2015) when extracting image representations by GoogleNet and achieve to date top-1 performance on COCO captioning Leaderboard. <s> BIB019 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Semantic-Concept-Based Image <s> A Semantic Compositional Network (SCN) is developed for image captioning, in which semantic concepts (i.e., tags) are detected from the image, and the probability of each tag is used to compose the parameters in a long short-term memory (LSTM) network. The SCN extends each weight matrix of the LSTM to an ensemble of tag-dependent weight matrices. The degree to which each member of the ensemble is used to generate an image caption is tied to the image-dependent probability of the corresponding tag. In addition to captioning images, we also extend the SCN to generate captions for video clips. We qualitatively analyze semantic composition in SCNs, and quantitatively evaluate the algorithm on three benchmark datasets: COCO, Flickr30k, and Youtube2Text. Experimental results show that the proposed method significantly outperforms prior state-of-the-art approaches, across multiple evaluation metrics. <s> BIB020 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Semantic-Concept-Based Image <s> Recently, there has been a lot of interest in automatically generating descriptions for an image. Most existing language-model based approaches for this task learn to generate an image description word by word in its original word order. However, for humans, it is more natural to locate the objects and their relationships first, and then elaborate on each object, describing notable attributes. We present a coarse-to-fine method that decomposes the original image description into a skeleton sentence and its attributes, and generates the skeleton sentence and attribute phrases separately. By this decomposition, our method can generate more accurate and novel descriptions than the previous state-of-the-art. Experimental results on the MS-COCO and a larger scale Stock3M datasets show that our algorithm yields consistent improvements across different evaluation metrics, especially on the SPICE metric, which has much higher correlation with human ratings than the conventional metrics. Furthermore, our algorithm can generate descriptions with varied length, benefiting from the separate control of the skeleton and attributes. This enables image description generation that better accommodates user preferences. <s> BIB021 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Semantic-Concept-Based Image <s> Much of the recent progress in Vision-to-Language problems has been achieved through a combination of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). This approach does not explicitly represent high-level semantic concepts, but rather seeks to progress directly from image features to text. In this paper we first propose a method of incorporating high-level concepts into the successful CNN-RNN approach, and show that it achieves a significant improvement on the state-of-the-art in both image captioning and visual question answering. We further show that the same mechanism can be used to incorporate external knowledge, which is critically important for answering high level visual questions. Specifically, we design a visual question answering model that combines an internal representation of the content of an image with information extracted from a general knowledge base to answer a broad range of image-based questions. It particularly allows questions to be asked where the image alone does not contain the information required to select the appropriate answer. Our final model achieves the best reported results for both image captioning and visual question answering on several of the major benchmark datasets. <s> BIB022
Captioning. Semantic-concept-based methods selectively attend to a set of semantic concept proposals extracted from the image. These concepts are then combined into hidden states and the outputs of recurrent neural networks. The methods in this category follow the following steps: (1) The CNN-based encoder is used to encode the image features and semantic concepts. A typical block diagram of this category is shown in Figure 8 . Karpathy et al. extended their method BIB005 in BIB008 . The later method can generate natural language descriptions for both images and their regions. This method employs a novel combination of CNN over the image regions, bidirectional recurrent neural networks over sentences, and a common multimodal embedding that associates the two modalities. It also demonstrates a multimodal recurrent neural network architecture that utilizes the resultant alignments to train the model for generating novel descriptions of image regions. In this method, dependency tree relations (DTRs) are used to train to map the sentence segments with the image regions that have a fixed window context. In contrast to their previous method, this method uses a bidirectional neural network to obtain word representations in the sentence. It considers contiguous fragments of sentences to align in the embedding space, which is more meaningful, interpretable, and not fixed in length. Generally an RNN considers the current word and the contexts from all the previously generated words for estimating a probability distribution of the next word in a sequence. However, this method extends it for considering the generative process on the content of an input image. This addition is simple, but it makes it very effective for generating novel image captions. Attributes of an image are considered as rich semantic cues. The method of Yao et al. BIB019 has different architectures to incorporate attributes with image representations. Mainly, two types of architectural representations are introduced here. In the first group, it inserts only attributes to the LSTM or image representations to the LSTM first and then attributes, and vice versa. In the second group, it can control the time step of the LSTM. It decides whether image representation and attributes will be input once or every time step. These variants of architectures are tested on the MS COCO dataset and common evaluation metrics. You et al. BIB016 proposed a semantic-attention-based image captioning method. The method provides a detailed, coherent description of semantically important objects. The top-down paradigms BIB009 BIB010 BIB008 BIB011 BIB006 BIB012 BIB013 are used for extracting visual features first and then convert them into words. In bottom-up approaches, BIB004 BIB001  BIB003 BIB007 BIB002 , visual concepts (e.g., regions, objects, and attributes) are extracted first from various aspects of an image and then combined. Fine details of an image are often very important for generating a description of an image. Top-down approaches have limitations in obtaining fine details of the image. Bottom-up approaches are capable of operating on any image resolution and therefore they can do work on fine details of the image. However, they have problems in formulating an end-to-end process. Therefore, semanticbased attention models applied both top-down and bottom-up approaches for generating image captions. In top-down approaches, the image features are obtained using the last 1,024-dimensional convolutional layer of the GoogleNet BIB014 CNN model. The visual concepts are collected using different nonparametric and parametric methods. The nearest neighbor image retrieval technique is used for computing nonparametric visual concepts. The fully convolutional network (FCN) BIB015 is used to learn attributes from local patches for parametric attribute prediction. Although Xu et al. BIB013 considered attention-based captioning, it works on fixed and predefined spatial location. However, this semantic-attention-based method can work on any resolution and any location of the image. Moreover, this method also considers a feedback process that accelerates to generate better image captions. Previous image captioning methods do not include high-level semantic concepts explicitly. However, Wu et al. BIB022 proposed a high-level semantic-concept-based image captioning. It uses an intermediate attribute prediction layer in a neural-network-based CNN-LSTM framework. First, attributes are extracted by a CNN-based classifier from training image captions. Then these attributes are used as high-level semantic concepts in generating semantically rich image captions. Recent semantic-concept-based image captioning methods BIB022 BIB016 applied the semantic concept detection process BIB017 to obtain explicit semantic concepts. They use these high-level semantic concepts in the CNN-LSTM-based encoder-decoder and achieve significant improvements in image captioning. However, they have problems in generating semantically sound captions. They cannot distribute semantic concepts evenly in the whole sentence. For example, Wu et al. BIB022 consider the initial state of the LSTM to add semantic concepts. Moreover, it encodes a visual feature vector or an inferred scene vector from the CNN and then feeds it to the LSTM for generating captions. However, Gan et al. BIB020 introduced a semantic compositional network (SCN) for image captioning. In this method, a semantic concept vector is constructed from all the probable concepts (called tags here) found in the image. This semantic vector has more potential than the visual feature vector and scene vector and can generate captions covering the overall meaning of the image. This is called a compositional network because it can compose most semantic concepts. Existing LSTM-based image captioning methods have limitations in generating a diverse set of captions because they have to predict the next word in a predefined word-by-word format. However, a combination of attributes and subjects and their relationship in a sentence irrespective of their location can generate a broad range of image captions. Wang et al. BIB021 proposed a method that locates the objects and their interactions first and then identifies and extracts the relevant attributes to generate image captions. The main aim of this method is to decompose the ground-truth image captions into two parts: skeleton sentences and attribute phrases. The method is also called Skeleton Key. The architecture of this method has ResNet BIB018 and two LSTMs called Skel-LSTM and Attr-LSTM. During training, skeleton sentences are trained by the Skel-LSTM network and attribute phrases are trained by the Attr-LSTM network. In the testing phase, skeleton sentences are generated first that contain the words for the main objects of the image and their relationships. Then these objects look back through the image again to obtain the relevant attributes. It is tested on the MS COCO dataset and a new Stock3M dataset and can generate more accurate and novel captions.