A Comprehensive Survey of Deep Learning for Image Captioning <s> COMPARISON ON BENCHMARK DATASETS AND COMMON <s> In this work we focus on the problem of image caption generation. We propose an extension of the long short term memory (LSTM) model, which we coin gLSTM for short. In particular, we add semantic information extracted from the image as extra input to each unit of the LSTM block, with the aim of guiding the model towards solutions that are more tightly coupled to the image content. Additionally, we explore different length normalization strategies for beam search to avoid bias towards short sentences. On various benchmark datasets such as Flickr8K, Flickr30K and MS COCO, we obtain results that are on par with or better than the current state-of-the-art. <s> BIB001 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> COMPARISON ON BENCHMARK DATASETS AND COMMON <s> Recent progress on automatic generation of image captions has shown that it is possible to describe the most salient information conveyed by images with accurate and meaningful sentences. In this paper, we propose an image caption system that exploits the parallel structures between images and sentences. In our model, the process of generating the next word, given the previously generated ones, is aligned with the visual perception experience where the attention shifting among the visual regions imposes a thread of visual ordering. This alignment characterizes the flow of "abstract meaning", encoding what is semantically shared by both the visual scene and the text description. Our system also makes another novel modeling contribution by introducing scene-specific contexts that capture higher-level semantic information encoded in an image. The contexts adapt language models for word generation to specific scene types. We benchmark our system and contrast to published results on several popular datasets. We show that using either region-based attention or scene-specific contexts improves systems without those components. Furthermore, combining these two modeling ingredients attains the state-of-the-art performance. <s> BIB002 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> COMPARISON ON BENCHMARK DATASETS AND COMMON <s> We propose a novel module, the reviewer module, to improve the encoder-decoder learning framework. The reviewer module is generic, and can be plugged into an existing encoder-decoder model. The reviewer module performs a number of review steps with attention mechanism on the encoder hidden states, and outputs a fact vector after each review step; the fact vectors are used as the input of the attention mechanism in the decoder. We show that the conventional encoder-decoders are a special case of our framework. Empirically, we show that our framework can improve over state-of-the-art encoder-decoder systems on the tasks of image captioning and source code captioning. <s> BIB003 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> COMPARISON ON BENCHMARK DATASETS AND COMMON <s> We propose "Areas of Attention", a novel attention-based model for automatic image captioning. Our approach models the dependencies between image regions, caption words, and the state of an RNN language model, using three pairwise interactions. In contrast to previous attention-based approaches that associate image regions only to the RNN state, our method allows a direct association between caption words and image regions. During training these associations are inferred from image-level captions, akin to weakly-supervised object detector training. These associations help to improve captioning by localizing the corresponding regions during testing. We also propose and compare different ways of generating attention areas: CNN activation grids, object proposals, and spatial transformers nets applied in a convolutional fashion. Spatial transformers give the best results. They allow for image specific attention areas, and can be trained jointly with the rest of the network. Our attention mechanism and spatial transformer attention areas together yield state-of-the-art results on the MSCOCO dataset.o meaningful latent semantic structure in the generated captions. <s> BIB004
EVALUATION METRICS While formal experimental evaluation was left out of the scope of this article, we present a brief analysis of the experimental results and the performance of various techniques as reported. We cover three sets of results: (1) We find that a number of methods use the first three datasets listed in Section 4.1 and a number of commonly used evaluation metrics to present the results. These results are shown in Table 3 . (2) A few methods fall into the following groups: attention-based and other deep-learningbased (reinforcement learning and GAN-based methods) image captioning. The results of such methods are shown in Tables 4 and 5, respectively. (3) We also list the methods that provide the top two results scored on each evaluation metric on the MS COCO dataset. These results are shown in Table 6 . BIB001 . The main reason behind this is that it uses an attention mechanism, which focuses only on relevant objects of the image. The semantic-conceptbased methods can generate semantically rich captions. Wu et al. BIB003 proposed a semanticconcept-based image captioning method. This method first predicts the attributes of different objects from the image and then adds these attributes with the captions that are semantically meaningful. In terms of performance, the method is superior to all the methods mentioned in Table 3 . Table 4 shows the results of attention-based based methods on the MS COCO dataset. Xu et al.'s stochastic hard attention produced better results than deterministic soft attention. However, these results were outperformed by the method of Jin et al. BIB002 , which can update its attention based on the scene-specific context. Wu et al. BIB003 and Pedersoli et al. BIB004 only show BLEU-4 and METEOR scores, which are higher than the aforementioned methods. The method of Wu et al. uses an attention mechanism with a review process. The review process checks the focused attention in every time step and updates it if necessary. This mechanism helps to achieve better results than the prior attention-based methods. Pedersoli et al. propose a different attention mechanism that maps the focused image regions directly with the caption words instead of the LSTM state. The behavior of this method drives it to achieve top performances among the mentioned attention-based methods in Table 4 .