A Comprehensive Survey of Deep Learning for Image Captioning <s> Compositional <s> The concept of maximum entropy can be traced back along multiple threads to Biblical times. Only recently, however, have computers become powerful enough to permit the widescale application of this concept to real world problems in statistical estimation and pattern recognition. In this paper, we describe a method for statistical modeling based on maximum entropy. We present a maximum-likelihood approach for automatically constructing maximum entropy models and describe how to implement this approach efficiently, using as examples several problems in natural language processing. <s> BIB001 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Compositional <s> Multiple-instance learning is a variation on supervised learning, where the task is to learn a concept given positive and negative bags of instances. Each bag may contain many instances, but a bag is labeled positive even if only one of the instances in it falls within the concept. A bag is labeled negative only if all the instances in it are negative. We describe a new general framework, called Diverse Density, for solving multiple-instance learning problems. We apply this framework to learn a simple description of a person from a series of images (bags) containing that person, to a stock selection problem, and to the drug activity prediction problem. <s> BIB002 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Compositional <s> Human evaluations of machine translation are extensive but expensive. Human evaluations can take months to finish and involve human labor that can not be reused. We propose a method of automatic machine translation evaluation that is quick, inexpensive, and language-independent, that correlates highly with human evaluation, and that has little marginal cost per run. We present this method as an automated understudy to skilled human judges which substitutes for them when there is need for quick or frequent evaluations. <s> BIB003 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Compositional <s> Often, the training procedure for statistical machine translation models is based on maximum likelihood or related criteria. A general problem of this approach is that there is only a loose relation to the final translation quality on unseen text. In this paper, we analyze various training criteria which directly optimize translation quality. These training criteria make use of recently proposed automatic evaluation metrics. We describe a new algorithm for efficient training an unsmoothed error count. We show that significantly better results can often be obtained if the final evaluation criterion is taken directly into account as part of the training procedure. <s> BIB004 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Compositional <s> We argue that the machine translation community is overly reliant on the Bleu machine translation evaluation metric. We show that an improved Bleu score is neither necessary nor sufficient for achieving an actual improvement in translation quality, and give two significant counterexamples to Bleu’s correlation with human judgments of quality. This offers new potential for research which was previously deemed unpromising by an inability to improve upon Bleu scores. <s> BIB005 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Compositional <s> This paper describes our submissions to the machine translation evaluation shared task in ACL WMT-08. Our primary submission is the Meteor metric tuned for optimizing correlation with human rankings of translation hypotheses. We show significant improvement in correlation as compared to the earlier version of metric which was tuned to optimized correlation with traditional adequacy and fluency judgments. We also describe m-bleu and m-ter, enhanced versions of two other widely used metrics bleu and ter respectively, which extend the exact word matching used in these metrics with the flexible matching based on stemming and Wordnet in Meteor. <s> BIB006 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Compositional <s> Freebase is a practical, scalable tuple database used to structure general human knowledge. The data in Freebase is collaboratively created, structured, and maintained. Freebase currently contains more than 125,000,000 tuples, more than 4000 types, and more than 7000 properties. Public read/write access to Freebase is allowed through an HTTP-based graph-query API using the Metaweb Query Language (MQL) as a data query and manipulation language. MQL provides an easy-to-use object-oriented interface to the tuple data in Freebase and is designed to facilitate the creation of collaborative, Web-based data-oriented applications. <s> BIB007 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Compositional <s> Adjusting photographs to obtain compelling renditions requires skill and time. Even contrast and brightness adjustments are challenging because they require taking into account the image content. Photographers are also known for having different retouching preferences. As the result of this complexity, rule-based, one-size-fits-all automatic techniques often fail. This problem can greatly benefit from supervised machine learning but the lack of training data has impeded work in this area. Our first contribution is the creation of a high-quality reference dataset. We collected 5,000 photos, manually annotated them, and hired 5 trained photographers to retouch each picture. The result is a collection of 5 sets of 5,000 example input-output pairs that enable supervised learning. We first use this dataset to predict a user's adjustment from a large training set. We then show that our dataset and features enable the accurate adjustment personalization using a carefully chosen set of training photos. Finally, we introduce difference learning: this method models and predicts difference between users. It frees the user from using predetermined photos for training. We show that difference learning enables accurate prediction using only a handful of examples. <s> BIB008 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Compositional <s> Deep Neural Networks (DNNs) are powerful models that have achieved excellent performance on difficult learning tasks. Although DNNs work well whenever large labeled training sets are available, they cannot be used to map sequences to sequences. In this paper, we present a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure. Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector. Our main result is that on an English to French translation task from the WMT-14 dataset, the translations produced by the LSTM achieve a BLEU score of 34.8 on the entire test set, where the LSTM's BLEU score was penalized on out-of-vocabulary words. Additionally, the LSTM did not have difficulty on long sentences. For comparison, a phrase-based SMT system achieves a BLEU score of 33.3 on the same dataset. When we used the LSTM to rerank the 1000 hypotheses produced by the aforementioned SMT system, its BLEU score increases to 36.5, which is close to the previous state of the art. The LSTM also learned sensible phrase and sentence representations that are sensitive to word order and are relatively invariant to the active and the passive voice. Finally, we found that reversing the order of the words in all source sentences (but not target sentences) improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier. <s> BIB009 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Compositional <s> This paper presents a novel approach for automatically generating image descriptions: visual detectors, language models, and multimodal similarity models learnt directly from a dataset of image captions. We use multiple instance learning to train visual detectors for words that commonly occur in captions, including many different parts of speech such as nouns, verbs, and adjectives. The word detector outputs serve as conditional inputs to a maximum-entropy language model. The language model learns from a set of over 400,000 image descriptions to capture the statistics of word usage. We capture global semantics by re-ranking caption candidates using sentence-level features and a deep multimodal similarity model. Our system is state-of-the-art on the official Microsoft COCO benchmark, producing a BLEU-4 score of 29.1%. When human judges compare the system captions to ones written by other people on our held-out test set, the system captions have equal or better quality 34% of the time. <s> BIB010 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Compositional <s> Automatically describing an image with a sentence is a long-standing challenge in computer vision and natural language processing. Due to recent progress in object detection, attribute classification, action recognition, etc., there is renewed interest in this area. However, evaluating the quality of descriptions has proven to be challenging. We propose a novel paradigm for evaluating image descriptions that uses human consensus. This paradigm consists of three main parts: a new triplet-based method of collecting human annotations to measure consensus, a new automated metric that captures consensus, and two new datasets: PASCAL-50S and ABSTRACT-50S that contain 50 sentences describing each image. Our simple metric captures human judgment of consensus better than existing metrics across sentences generated by various sources. We also evaluate five state-of-the-art image description approaches using this new protocol and provide a benchmark for future comparisons. A version of CIDEr named CIDEr-D is available as a part of MS COCO evaluation server to enable systematic evaluation and benchmarking. <s> BIB011 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Compositional <s> Two recent approaches have achieved state-of-the-art results in image captioning. The first uses a pipelined process where a set of candidate words is generated by a convolutional neural network (CNN) trained on images, and then a maximum entropy (ME) language model is used to arrange these words into a coherent sentence. The second uses the penultimate activation layer of the CNN as input to a recurrent neural network (RNN) that then generates the caption sequence. In this paper, we compare the merits of these different language modeling approaches for the first time by using the same state-ofthe-art CNN as input. We examine issues in the different approaches, including linguistic irregularities, caption repetition, and data set overlap. By combining key aspects of the ME and RNN methods, we achieve a new record performance over previously published results on the benchmark COCO dataset. However, the gains we see in BLEU do not translate to human judgments. <s> BIB012 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Compositional <s> Current research into automatic generation of semantic descriptions centers mainly on improving the annotation accuracy for individual tag or attributes. In this paper, we focus on the generation of more informative descriptions for images. We proposes to generate layered, semantically meaningful descriptions and create summaries of key aspects of the data from the component detectors. In particular, the output descriptions include superclass, class, attributes, and the location of the area of the object which may interest users. We propose to integrate ROI (Region of Interest) identification and hierarchical semantic elements detection into a joint framework. The joint optimization of the ROI localizer and the hierarchical concept detection make them mutually beneficial and reciprocal. In this way, we create a discriminative image description generation framework based on a tightly coupled multi-layer optimization. The output descriptions contain richer information of the image content with layered contextual information, thereby enabling better management and usage of image data. Experiments on two public open benchmark datasets demonstrate that the proposed method obtains state of the art performance. <s> BIB013 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Compositional <s> We present an image caption system that addresses new challenges of automatically describing images in the wild. The challenges include generating high quality caption with respect to human judgments, out-of-domain data handling, and low latency required in many applications. Built on top of a state-of-the-art framework, we developed a deep vision model that detects a broad range of visual concepts, an entity recognition model that identifies celebrities and landmarks, and a confidence model for the caption output. Experimental results show that our caption engine outperforms previous state-of-the-art systems significantly on both in-domain dataset (i.e. MS COCO) and out-of-domain datasets. We also make the system publicly accessible as a part of the Microsoft Cognitive Services. <s> BIB014 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Compositional <s> Generating semantic description draws increasing attention recently. Describing objects with adaptive adjunct words make the sentence more informative. In this paper, we focus on the generation of descriptions for images according to the structural words we have generated such as a tetrad of . We propose to use deep machine translation method to generate semantically meaningful descriptions. In particular, the description is composed of objects with appropriate adjunct words, corresponding activities and scene. We propose to use a multi-task method to generate structural words. Taking these words sequence as source language, we train a LSTM encoder-decoder machine translation model to output the target language. Experiments on the benchmark datasets demonstrate our method has better performance than state-of-the-art methods of image caption in terms of language generation metrics. <s> BIB015 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Compositional <s> The models based on deep convolutional networks and recurrent neural networks have dominated in recent image caption generation tasks. Performance and complexity are still eternal topic. Inspired by recent work, by combining the advantages of simple RNN and LSTM, we present a novel parallel-fusion RNN-LSTM architecture, which obtains better results than a dominated one and improves the efficiency as well. The proposed approach divides the hidden units of RNN into several same-size parts, and lets them work in parallel. Then, we merge their outputs with corresponding ratios to generate final results. Moreover, these units can be different types of RNNs, for instance, a simple RNN and a LSTM. By training normally using NeuralTalk1 platform on Flickr8k dataset, without additional training data, we get better results than that of dominated structure and particularly, the proposed model surpass GoogleNIC in image caption generation. <s> BIB016 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Compositional <s> We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0%, respectively, which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overfitting in the fully connected layers we employed a recently developed regularization method called "dropout" that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry. <s> BIB017
Architecture-Based Image Captioning. Compositional architecture-based methods are composed of several independent functional building blocks: First, a CNN is used to extract the semantic concepts from the image. Then a language model is used to generate a set of candidate captions. In generating the final caption, these candidate captions are reranked using a deep multimodal similarity model. A typical method of this category maintains the following steps: (1) Image features are obtained using a CNN. (3) Multiple captions are generated by a language model using the information of Step 1 and Step 2. (4) The generated captions are reranked using a deep multimodal similarity model to select high-quality image captions. A common block diagram of compositional network-based image captioning methods is given in Figure 6 . Fang et al. BIB010 introduced generation-based image captioning. It uses visual detectors, a language model, and a multimodal similarity model to train the model on an image captioning dataset. Image captions can contain nouns, verbs, and adjectives. A vocabulary is formed using the 1,000 most common words from the training captions. The system works with the image subregions rather that the full image. Convolutional neural networks (both AlexNet BIB017 and VGG16Net) are used for extracting features for the subregions of an image. The features of subregions are mapped with the words of the vocabulary that are likely to be contained in the image captions. Multiple instance learning (MIL) BIB002 is used to train the model for learning discriminative visual signatures of each word. A maximum entropy (ME) BIB001 language model is used for generating image captions from these words. Generated captions are ranked by a linear weighting of sentence features. Minimum error rate training (MERT) BIB004 is used to learn these weights. Similarities between image and sentence can be easily measured using a common vector representation. Image and sentence fragments are mapped with the common vector representation by a deep multimodal similarity model (DMSM). It achieves a significant improvement in choosing high-quality image captions. Until now, a significant number of methods have achieved satisfactory progress in generating image captions. The methods use training and testing samples from the same domain. Therefore, there is no certainty that these methods can perform well in open-domain images. Moreover, they are only good at recognizing generic visual content. There are certain key entities such as celebrities and landmarks that are out of their scope. The generated captions of these methods are evaluated on automatic metrics such as BLEU BIB003 , METEOR BIB006 , and CIDEr BIB011 . These evaluation metrics have already shown good results on these methods. However, in terms of performance, there exists a large gap between the evaluation of the metrics and human judgment of evaluation BIB005 BIB012  . If it is considered real-life entity information, the performance could be weaker. However, Tran et al. BIB014 introduced a different image captioning method. This method is capable of generating image captions even for open-domain images. It can detect a diverse set of visual concepts and generate captions for celebrities and landmarks. It uses an external knowledge base, Freebase BIB007 , in recognizing a broad range of entities such as celebrities and landmarks. A series of human judgments are applied for evaluating the performances of generated captions. In experiments, it uses three datasets: MS COCO, Adobe-MIT FiveK BIB008 , and images from Instagram. The images of the MS COCO dataset were collected from the same domain, but the images of other datasets were chosen from an open domain. The method achieves notable performances especially on the challenging Instagram dataset. Ma et al. BIB015 proposed another compositional network-based image captioning method. This method uses structural words <object, attribute, activity, scene> to generate semantically meaningful descriptions. It also uses a multitask method similar to the multiple instance learning method BIB010 and multilayer optimization method BIB013 to generate structural words. An LSTM encoderdecoder-based machine translation method BIB009 is then used to translate the structural words into image captions. Wang et al. BIB016 proposed a parallel-fusion RNN-LSTM architecture for image caption generation. The architecture of the method divides the hidden units of RNN and LSTM into a number of same-size parts. The parts work in parallel with corresponding ratios to generate image captions.