A Comprehensive Survey of Deep Learning for Image Captioning <s> Attention-Based Image <s> We describe latent Dirichlet allocation (LDA), a generative probabilistic model for collections of discrete data such as text corpora. LDA is a three-level hierarchical Bayesian model, in which each item of a collection is modeled as a finite mixture over an underlying set of topics. Each topic is, in turn, modeled as an infinite mixture over an underlying set of topic probabilities. In the context of text modeling, the topic probabilities provide an explicit representation of a document. We present efficient approximate inference techniques based on variational methods and an EM algorithm for empirical Bayes parameter estimation. We report results in document modeling, text classification, and collaborative filtering, comparing to a mixture of unigrams model and the probabilistic LSI model. <s> BIB001 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Attention-Based Image <s> In this work, we investigate eye movement analysis as a new sensing modality for activity recognition. Eye movement data were recorded using an electrooculography (EOG) system. We first describe and evaluate algorithms for detecting three eye movement characteristics from EOG signals-saccades, fixations, and blinks-and propose a method for assessing repetitive patterns of eye movements. We then devise 90 different features based on these characteristics and select a subset of them using minimum redundancy maximum relevance (mRMR) feature selection. We validate the method using an eight participant study in an office environment using an example set of five activity classes: copying a text, reading a printed paper, taking handwritten notes, watching a video, and browsing the Web. We also include periods with no specific activity (the NULL class). Using a support vector machine (SVM) classifier and person-independent (leave-one-person-out) training, we obtain an average precision of 76.1 percent and recall of 70.5 percent over all classes and participants. The work demonstrates the promise of eye-based activity recognition (EAR) and opens up discussion on the wider applicability of EAR to other activities that are difficult, or even impossible, to detect using common sensing modalities. <s> BIB002 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Attention-Based Image <s> We present a probabilistic generative model for simultaneously recognizing daily actions and predicting gaze locations in videos recorded from an egocentric camera. We focus on activities requiring eye-hand coordination and model the spatio-temporal relationship between the gaze point, the scene objects, and the action label. Our model captures the fact that the distribution of both visual features and object occurrences in the vicinity of the gaze point is correlated with the verb-object pair describing the action. It explicitly incorporates known properties of gaze behavior from the psychology literature, such as the temporal delay between fixation and manipulation events. We present an inference method that can predict the best sequence of gaze locations and the associated action label from an input sequence of images. We demonstrate improvements in action recognition rates and gaze prediction accuracy relative to state-of-the-art methods, on two new datasets that contain egocentric videos of daily activities and gaze. <s> BIB003 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Attention-Based Image <s> Large Convolutional Network models have recently demonstrated impressive classification performance on the ImageNet benchmark Krizhevsky et al. [18]. However there is no clear understanding of why they perform so well, or how they might be improved. In this paper we explore both issues. We introduce a novel visualization technique that gives insight into the function of intermediate feature layers and the operation of the classifier. Used in a diagnostic role, these visualizations allow us to find model architectures that outperform Krizhevsky et al on the ImageNet classification benchmark. We also perform an ablation study to discover the performance contribution from different model layers. We show our ImageNet model generalizes well to other datasets: when the softmax classifier is retrained, it convincingly beats the current state-of-the-art results on Caltech-101 and Caltech-256 datasets. <s> BIB004 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Attention-Based Image <s> Eye movement studies have confirmed that overt attention is highly biased towards faces and text regions in images. In this paper we explore a novel problem of predicting face and text regions in images using eye tracking data from multiple subjects. The problem is challenging as we aim to predict the semantics (face/text/background) only from eye tracking data without utilizing any image information. The proposed algorithm spatially clusters eye tracking data obtained in an image into different coherent groups and subsequently models the likelihood of the clusters containing faces and text using a fully connected Markov Random Field (MRF). Given the eye tracking data from a test image, it predicts potential face/head (humans, dogs and cats) and text locations reliably. Furthermore, the approach can be used to select regions of interest for further analysis by object detectors for faces and text. The hybrid eye position/object detector approach achieves better detection performance and reduced computation time compared to using only the object detection algorithm. We also present a new eye tracking dataset on 300 images selected from ICDAR, Street-view, Flickr and Oxford-IIIT Pet Dataset from 15 subjects. <s> BIB005 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Attention-Based Image <s> We posit that a person’s gaze behavior while freely viewing a scene contains an abundance of information, not only about their intent and what they consider to be important in the scene, but also about the scene’s content. Experiments are reported, using two popular image datasets from computer vision, that explore the relationship between the fixations that people make during scene viewing, how they describe the scene, and automatic detection predictions of object categories in the scene. From these exploratory analyses, we then combine human behavior with the outputs of current visual recognition methods to build prototype human in the loop applications for gaze-enabled object detection and scene annotation. <s> BIB006 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Attention-Based Image <s> WHAT DOES IT MEAN TO “UNDERSTAND” A SCENE? Look very briefly at the picture on the next page (Figure 1); what do you see? Decades of research has shown that it is possible to extract the gist of a scene very quickly (Potter, 1975; Friedman, 1979); even from the briefest of glances you could probably classify this scene as some sort of race. Other research has shown that some categories of objects, people and animals in particular, can also be detected from very brief exposures (Thorpe et al., 1996; Joubert et al., 2007). It may even be possible to infer from the features of a scene some properties of these objects, such as a person’s gender and perhaps their emotion (Schyns and Oliva, 1999). Actions can also be characterized by relatively simple features (see Kozlowski and Cutting, 1977 and Wang et al., 2009, for two very different approaches), with the pose adopted by someone running being especially distinctive. So in addition to detecting that there were people in this scene, you could probably also discern that these people were women and that they were running. Combining these pieces of information might therefore have led you to a simple “gist” level of interpretation—a women’s track meet. But scene understanding doesn’t stop at gist, and there were other things happening in this scene. Closer scrutiny reveals that one of the runners had fallen and that the others were trying to avoid tripping over her. These details may have gone undetected from a brief exposure for a variety of reasons. When a person falls their pose becomes highly variable, and certainly deviates from poses that we have associated with typical human activities. Moreover, this particular fall resulted in the loss of this person’s face from view. These factors would make it harder to recognize this pattern of pixels as a person, which would obviously affect whether these details would be included in a scene interpretation following a brief exposure. “Avoidance” is also a far more difficult action to detect than running, characterized by a relatively subtle shift in body posture and a slightly exaggerating leaping stride by the runner. Had these fallen and avoidance events been detected they would almost certainly have been included in the scene interpretation, but the point is that it is unsurprising that they were not. And finally, if you had noticed that all of the runners had one prosthetic leg, this critical information would be expected to fundamentally change your understanding of this scene—suddenly it tells a story about a special race for women amputees. This example illustrates the fact that scene understanding exists on a continuum. At one end is a very fast and seemingly effortless extraction of the scene’s gist—often just its category name. At the other end is the slower and often effortful attachment of deeper meaning to the scene. Although these different interpretations likely engage different underlying processes and might therefore be given different names, for the purpose of this paper I will adopt the lay person’s definition of scene understanding—what is the scene about? What is the story that it is trying to tell? <s> BIB007 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Attention-Based Image <s> Deep Neural Networks (DNNs) are powerful models that have achieved excellent performance on difficult learning tasks. Although DNNs work well whenever large labeled training sets are available, they cannot be used to map sequences to sequences. In this paper, we present a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure. Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector. Our main result is that on an English to French translation task from the WMT-14 dataset, the translations produced by the LSTM achieve a BLEU score of 34.8 on the entire test set, where the LSTM's BLEU score was penalized on out-of-vocabulary words. Additionally, the LSTM did not have difficulty on long sentences. For comparison, a phrase-based SMT system achieves a BLEU score of 33.3 on the same dataset. When we used the LSTM to rerank the 1000 hypotheses produced by the aforementioned SMT system, its BLEU score increases to 36.5, which is close to the previous state of the art. The LSTM also learned sensible phrase and sentence representations that are sensitive to word order and are relatively invariant to the active and the passive voice. Finally, we found that reversing the order of the words in all source sentences (but not target sentences) improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier. <s> BIB008 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Attention-Based Image <s> We present a new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding. This is achieved by gathering images of complex everyday scenes containing common objects in their natural context. Objects are labeled using per-instance segmentations to aid in precise object localization. Our dataset contains photos of 91 objects types that would be easily recognizable by a 4 year old. With a total of 2.5 million labeled instances in 328k images, the creation of our dataset drew upon extensive crowd worker involvement via novel user interfaces for category detection, instance spotting and instance segmentation. We present a detailed statistical analysis of the dataset in comparison to PASCAL, ImageNet, and SUN. Finally, we provide baseline performance analysis for bounding box and segmentation detection results using a Deformable Parts Model. <s> BIB009 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Attention-Based Image <s> Training an object class detector typically requires a large set of images annotated with bounding-boxes, which is expensive and time consuming to create. We propose novel approach to annotate object locations which can substantially reduce annotation time. We first track the eye movements of annotators instructed to find the object and then propose a technique for deriving object bounding-boxes from these fixations. To validate our idea, we collected eye tracking data for the trainval part of 10 object classes of Pascal VOC 2012 (6,270 images, 5 observers). Our technique correctly produces bounding-boxes in 50%of the images, while reducing the total annotation time by factor 6.8× compared to drawing bounding-boxes. Any standard object class detector can be trained on the bounding-boxes predicted by our model. Our large scale eye tracking dataset is available at groups.inf.ed.ac.uk/calvin/eyetrackdataset/ . <s> BIB010 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Attention-Based Image <s> Inspired by recent work in machine translation and object detection, we introduce an attention based model that automatically learns to describe the content of images. We describe how we can train this model in a deterministic manner using standard backpropagation techniques and stochastically by maximizing a variational lower bound. We also show through visualization how the model is able to automatically learn to fix its gaze on salient objects while generating the corresponding words in the output sequence. We validate the use of attention with state-of-the-art performance on three benchmark datasets: Flickr9k, Flickr30k and MS COCO. <s> BIB011 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Attention-Based Image <s> Recent progress on automatic generation of image captions has shown that it is possible to describe the most salient information conveyed by images with accurate and meaningful sentences. In this paper, we propose an image caption system that exploits the parallel structures between images and sentences. In our model, the process of generating the next word, given the previously generated ones, is aligned with the visual perception experience where the attention shifting among the visual regions imposes a thread of visual ordering. This alignment characterizes the flow of "abstract meaning", encoding what is semantically shared by both the visual scene and the text description. Our system also makes another novel modeling contribution by introducing scene-specific contexts that capture higher-level semantic information encoded in an image. The contexts adapt language models for word generation to specific scene types. We benchmark our system and contrast to published results on several popular datasets. We show that using either region-based attention or scene-specific contexts improves systems without those components. Furthermore, combining these two modeling ingredients attains the state-of-the-art performance. <s> BIB012 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Attention-Based Image <s> The Flickr30k dataset has become a standard benchmark for sentence-based image description. This paper presents Flickr30k Entities, which augments the 158k captions from Flickr30k with 244k coreference chains linking mentions of the same entities in images, as well as 276k manually annotated bounding boxes corresponding to each entity. Such annotation is essential for continued progress in automatic image description and grounded language understanding. We present experiments demonstrating the usefulness of our annotations for text-to-image reference resolution, or the task of localizing textual entity mentions in an image, and for bidirectional image-sentence retrieval. These experiments confirm that we can further improve the accuracy of state-of-the-art retrieval methods by training with explicit region-to-phrase correspondence, but at the same time, they show that accurately inferring this correspondence given an image and a caption remains really challenging. <s> BIB013 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Attention-Based Image <s> Previous work on predicting the target of visual search from human fixations only considered closed-world settings in which training labels are available and predictions are performed for a known set of potential targets. In this work we go beyond the state of the art by studying search target prediction in an open-world setting in which we no longer assume that we have fixation data to train for the search targets. We present a dataset containing fixation data of 18 users searching for natural images from three image categories within synthesised image collages of about 80 images. In a closed-world baseline experiment we show that we can predict the correct target image out of a candidate set of five images. We then present a new problem formulation for search target prediction in the open-world setting that is based on learning compatibilities between fixations and potential targets. <s> BIB014 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Attention-Based Image <s> Visual attention is a crucial indicator of the relative importance of objects in visual scenes to human viewers. In this paper, we propose an algorithm to extract objects which attract visual attention from videos. As human attention is naturally biased towards high level semantic objects in visual scenes, this information can be valuable to extract salient objects. The proposed algorithm extracts dominant visual tracks using eye tracking data from multiple subjects on a video sequence by a combination of mean-shift clustering and Hungarian algorithm. These visual tracks guide a generic object search algorithm to get candidate object locations and extents in every frame. Further, we propose a novel multiple object extraction algorithm by constructing a spatio-temporal mixed graph over object candidates. Bounding box based object extraction inference is performed using binary linear integer programming on a cost function defined over the graph. Finally, the object boundaries are refined using grabcut segmentation. The proposed technique outperforms state-of-the-art video segmentation using eye tracking prior and obtains favorable object extraction over algorithms which do not utilize eye tracking data. <s> BIB015 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Attention-Based Image <s> Saliency in Context (SALICON) is an ongoing effort that aims at understanding and predicting visual attention. This paper presents a new method to collect large-scale human data during natural explorations on images. While current datasets present a rich set of images and task-specific annotations such as category labels and object segments, this work focuses on recording and logging how humans shift their attention during visual exploration. The goal is to offer new possibilities to (1) complement task-specific annotations to advance the ultimate goal in visual understanding, and (2) understand visual attention and learn saliency models, all with human attentional data at a much larger scale. We designed a mouse-contingent multi-resolutional paradigm based on neurophysiological and psychophysical studies of peripheral vision, to simulate the natural viewing behavior of humans. The new paradigm allowed using a general-purpose mouse instead of an eye tracker to record viewing behaviors, thus enabling large-scale data collection. The paradigm was validated with controlled laboratory as well as large-scale online data. We report in this paper a proof-of-concept SALICON dataset of human “free-viewing” data on 10,000 images from the Microsoft COCO (MS COCO) dataset with rich contextual information. We evaluated the use of the collected data in the context of saliency prediction, and demonstrated them a good source as ground truth for the evaluation of saliency algorithms. <s> BIB016 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Attention-Based Image <s> We propose a novel module, the reviewer module, to improve the encoder-decoder learning framework. The reviewer module is generic, and can be plugged into an existing encoder-decoder model. The reviewer module performs a number of review steps with attention mechanism on the encoder hidden states, and outputs a fact vector after each review step; the fact vectors are used as the input of the attention mechanism in the decoder. We show that the conventional encoder-decoders are a special case of our framework. Empirically, we show that our framework can improve over state-of-the-art encoder-decoder systems on the tasks of image captioning and source code captioning. <s> BIB017 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Attention-Based Image <s> Attention mechanisms have recently been introduced in deep learning for various tasks in natural language processing and computer vision. But despite their popularity, the "correctness" of the implicitly-learned attention maps has only been assessed qualitatively by visualization of several examples. In this paper we focus on evaluating and improving the correctness of attention in neural image captioning models. Specifically, we propose a quantitative evaluation metric for the consistency between the generated attention maps and human annotations, using recently released datasets with alignment between regions in images and entities in captions. We then propose novel models with different levels of explicit supervision for learning attention maps during training. The supervision can be strong when alignment between regions and caption entities are available, or weak when only object segments and categories are provided. We show on the popular Flickr30k and COCO datasets that introducing supervision of attention maps during training solidly improves both attention correctness and caption quality, showing the promise of making machine perception more human-like. <s> BIB018 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Attention-Based Image <s> Gaze reflects how humans process visual scenes and is therefore increasingly used in computer vision systems. Previous works demonstrated the potential of gaze for object-centric tasks, such as object localization and recognition, but it remains unclear if gaze can also be beneficial for scene-centric tasks, such as image captioning. We present a new perspective on gaze-assisted image captioning by studying the interplay between human gaze and the attention mechanism of deep neural networks. Using a public large-scale gaze dataset, we first assess the relationship between state-of-the-art object and scene recognition models, bottom-up visual saliency, and human gaze. We then propose a novel split attention model for image captioning. Our model integrates human gaze information into an attention-based long short-term memory architecture, and allows the algorithm to allocate attention selectively to both fixated and non-fixated image regions. Through evaluation on the COCO/SALICON datasets we show that our method improves image captioning performance and that gaze can complement machine attention for semantic scene understanding tasks. <s> BIB019 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Attention-Based Image <s> We propose "Areas of Attention", a novel attention-based model for automatic image captioning. Our approach models the dependencies between image regions, caption words, and the state of an RNN language model, using three pairwise interactions. In contrast to previous attention-based approaches that associate image regions only to the RNN state, our method allows a direct association between caption words and image regions. During training these associations are inferred from image-level captions, akin to weakly-supervised object detector training. These associations help to improve captioning by localizing the corresponding regions during testing. We also propose and compare different ways of generating attention areas: CNN activation grids, object proposals, and spatial transformers nets applied in a convolutional fashion. Spatial transformers give the best results. They allow for image specific attention areas, and can be trained jointly with the rest of the network. Our attention mechanism and spatial transformer attention areas together yield state-of-the-art results on the MSCOCO dataset.o meaningful latent semantic structure in the generated captions. <s> BIB020 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Attention-Based Image <s> Attention-based neural encoder-decoder frameworks have been widely adopted for image captioning. Most methods force visual attention to be active for every generated word. However, the decoder likely requires little to no visual information from the image to predict non-visual words such as the and of. Other words that may seem visual can often be predicted reliably just from the language model e.g., sign after behind a red stop or phone following talking on a cell. In this paper, we propose a novel adaptive attention model with a visual sentinel. At each time step, our model decides whether to attend to the image (and if so, to which regions) or to the visual sentinel. The model decides whether to attend to the image and where, in order to extract meaningful information for sequential word generation. We test our method on the COCO image captioning 2015 challenge dataset and Flickr30K. Our approach sets the new state-of-the-art by a significant margin. <s> BIB021 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Attention-Based Image <s> Visual attention has been successfully applied in structural prediction tasks such as visual captioning and question answering. Existing visual attention models are generally spatial, i.e., the attention is modeled as spatial probabilities that re-weight the last conv-layer feature map of a CNN encoding an input image. However, we argue that such spatial attention does not necessarily conform to the attention mechanism — a dynamic feature extractor that combines contextual fixations over time, as CNN features are naturally spatial, channel-wise and multi-layer. In this paper, we introduce a novel convolutional neural network dubbed SCA-CNN that incorporates Spatial and Channel-wise Attentions in a CNN. In the task of image captioning, SCA-CNN dynamically modulates the sentence generation context in multi-layer feature maps, encoding where (i.e., attentive spatial locations at multiple layers) and what (i.e., attentive channels) the visual attention is. We evaluate the proposed SCA-CNN architecture on three benchmark image captioning datasets: Flickr8K, Flickr30K, and MSCOCO. It is consistently observed that SCA-CNN significantly outperforms state-of-the-art visual attention-based image captioning methods. <s> BIB022 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Attention-Based Image <s> To bridge the gap between humans and machines in image understanding and describing, we need further insight into how people describe a perceived scene. In this paper, we study the agreement between bottom-up saliency-based visual attention and object referrals in scene description constructs. We investigate the properties of human-written descriptions and machine-generated ones. We then propose a saliency-boosted image captioning model in order to investigate benefits from low-level cues in language models. We learn that (1) humans mention more salient objects earlier than less salient ones in their descriptions, (2) the better a captioning model performs, the better attention agreement it has with human descriptions, (3) the proposed saliencyboosted model, compared to its baseline form, does not improve significantly on the MS COCO database, indicating explicit bottom-up boosting does not help when the task is well learnt and tuned on a data, (4) a better generalization is, however, observed for the saliency-boosted model on unseen data. <s> BIB023 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Attention-Based Image <s> State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet [1] and Fast R-CNN [2] have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features—using the recently popular terminology of neural networks with ’attention’ mechanisms, the RPN component tells the unified network where to look. For the very deep VGG-16 model [3] , our detection system has a frame rate of 5 fps ( including all steps ) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007, 2012, and MS COCO datasets with only 300 proposals per image. In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the foundations of the 1st-place winning entries in several tracks. Code has been made publicly available. <s> BIB024 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Attention-Based Image <s> We address personalization issues of image captioning, which have not been discussed yet in previous research. For a query image, we aim to generate a descriptive sentence, accounting for prior knowledge such as the users active vocabularies in previous documents. As applications of personalized image captioning, we tackle two post automation tasks: hashtag prediction and post generation, on our newly collected Instagram dataset, consisting of 1.1M posts from 6.3K users. We propose a novel captioning model named Context Sequence Memory Network (CSMN). Its unique updates over previous memory network models include (i) exploiting memory as a repository for multiple types of context information, (ii) appending previously generated words into memory to capture long-term information without suffering from the vanishing gradient problem, and (iii) adopting CNN memory structure to jointly represent nearby ordered memory slots for better context understanding. With quantitative evaluation and user studies via Amazon Mechanical Turk, we show the effectiveness of the three novel features of CSMN and its performance enhancement for personalized image captioning over state-of-the-art captioning models. <s> BIB025
Captioning. Neural encoder-decoder-based approaches were mainly used in machine translation BIB008 . Following these trends, they have also been used for the task of image captioning and found very effective. In image captioning, a CNN is used as an encoder to extract the visual features from the input image and an RNN is used as a decoder to convert this representation word by word into a natural language description of the image. However, these methods are unable to analyze the image over time while they generate the descriptions for the image. In addition to this, the methods do not consider the spatial aspects of the image that are relevant to the parts of the image captions. Instead, they generate captions considering the scene as a whole. Attention-based mechanisms are becoming increasingly popular in deep learning because they can address these limitations. They can dynamically focus on the various parts of the input image while the output sequences are being produced. A typical method of this category adopts the following steps: (1) Image information is obtained based on the whole scene by a CNN. (2) The language generation phase generates words or phrases based on the output of Step 1. (3) Salient regions of the given image are focused in each time step of the language generation model based on generated words or phrases. (4) Captions are updated dynamically until the end state of the language generation model. A block diagram of the attention-based image captioning method is shown in Figure 7 . Xu et al. BIB011 were the first to introduce an attention-based image captioning method. The method describes the salient contents of an image automatically. The main difference between the attention-based methods and other methods is that they can concentrate on the salient parts of the image and generate the corresponding words at the same time. This method applies two different techniques: stochastic hard attention and deterministic soft attention to generate attention. Most CNN-based approaches use the top layer of ConvNet for extracting information of the salient objects from the image. A drawback of these techniques is that they may lose certain information that is useful to generate detailed captions. In order to preserve the information, the attention method uses features from the lower convolutional layer instead of the fully connected layer. Jin et al. BIB012 proposed another attention-based image captioning method. This method is capable of extracting the flow of abstract meaning based on the semantic relationship between visual information and textual information. It can also obtain higher-level semantic information by proposing a scene-specific context. The main difference between this method and other attentionbased methods is that it introduces multiple visual regions of an image at multiple scales. This technique can extract proper visual information of a particular object. For extracting a scenespecific context, it first uses the latent Dirichlet allocation (LDA) BIB001 for generating a dictionary from all the captions of the dataset. Then a multilayer perceptron is used to predict a topic vector for every image. A scene-factored LSTM that has two stacked layers is used to generate a description for the overall context of the image. Wu et al. BIB017 proposed a review-based attention method for image captioning. It introduces a review model that can perform multiple review steps with attention on CNN hidden states. The output of the CNN is a number of fact vectors that can obtain the global facts of the image. The vectors are given as input to the attention mechanism of the LSTM. For example, a reviewer module can first review: what are the objects in the image? Then it can review the relative positions of the objects and another review can extract the information of the overall context of the image. This information is passed to the decoder to generate image captions. Pedersoli et al. BIB020 proposed an area-based attention mechanism for image captioning. Previous attention-based methods map image regions only to the state of the RNN language model. However, this approach associates image regions with caption words given the RNN state. It can predict the next caption word and corresponding image region in each time step of RNN. It is capable of predicting the next word as well as corresponding image regions in each time step of RNN for generating image captions. In order to find the areas of attention, previous attention-based image caption methods used either the position of the CNN activation grid or object proposals. In contrast, this method uses an end-to-end trainable convolutional spatial transformer along with a CNN activation gird and object proposal methods. A combination of these techniques helps this method to compute image-adaptive areas of attention. In experiments, the method shows that this new attention mechanism together with the spatial transformer network can produce high-quality image captions. Lu et al. BIB021 proposed another attention-based image captioning method. The method is based on an adaptive attention model with a visual sentinel. Current attention-based image captioning methods focus on the image in every time step of RNN. However, there are some words or phrases (e.g., "a," "of") that do not need to attend visual signals. Moreover, these unnecessary visual signals could affect the caption generation process and degrade the overall performance. Therefore, their proposed method can determine when it will focus on an image region and when it will just focus on a language generation model. Once it determines to focus on the image, then it must have to choose the spatial location of the image. The first contribution of this method is to introduce a novel spatial attention method that can compute spatial features from the image. Then, in their adaptive attention method, they introduced a new LSTM extension. Generally, an LSTM works as a decoder that can produce a hidden state at every time step. However, this extension is capable of producing an additional visual sentinel that provides a fall-back option to the decoder. It also has a sentinel gate that can control how much information the decoder will get from the image. While attention-based methods look to find the different areas of the image at the time of generating words or phrases for image captions, the attention maps generated by these methods cannot always correspond to the proper region of the image. It can affect the performance of image caption generation. Liu et al. BIB018 proposed a method for neural image captioning. This method can evaluate and correct the attention map at time steps. Correctness means making a consistent map between image regions and generated words. In order to achieve these goals, this method introduced a quantitative evaluation metric to compute the attention maps. It uses the Flickr30K entity dataset BIB013 and the MS COCO BIB009 dataset for measuring both a ground-truth attention map and semantic labelings of image regions. In order to learn a better attention function, it proposed a supervised attention model. Two types of supervised attention models are used here: strong supervision with alignment annotation and weak supervision with semantic labeling. In the strong supervision with alignment annotation model, it can directly map ground-truth words to a region. However, ground-truth alignment is not always possible because collecting and annotating data is often very expensive. Weak supervision is performed to use bounding box or segmentation masks on the MS COCO dataset. In experiments, the method shows that the supervised attention model performs better in mapping attention as well as image captioning. Chen et al. BIB022 proposed another attention-based image captioning method. This method considers both spatial and channel-wise attentions to compute an attention map. The existing attention-based image captioning methods only consider spatial information for generating an attention map. A common drawback of these spatial attention methods is that they compute weighted pooling only on attentive feature maps. As a result, these methods lose the spatial information gradually. Moreover, they use the spatial information only from the last conv-layer of the CNN. The receptive field regions of this layer are quite large, which limits the gap between the regions. Therefore, they do not get significant spatial attention for an image. However, in this method, CNN features are extracted not only from spatial locations but also from different channels and multiple layers. Therefore, it gets significant spatial attention. In addition to this, in this method, each filter of a convolutional layer acts as a semantic detector BIB004 , while other methods use external sources for obtaining semantic information. In order to reduce the gap between the human-generated description and machine-generated description, Tavakoli et al. BIB023 introduced an attention-based image captioning method. This is a bottom-up saliency-based attention model that can take advantage of comparisons with other attention-based image captioning methods. It found that humans first describe the more important objects before less important ones. It also shows that the method performs better on unseen data. Most previous image captioning methods applied a top-down approach for constructing a visual attention map. These mechanisms typically focused on some selective regions obtained from the output of one or two layers of a CNN. The input regions are of the same size and have the same shape of receptive field. This approach has little consideration for the content of the image. However, the method of Anderson et al.  applied both top-down and bottom-up approaches. The bottom-up attention mechanism uses Faster R-CNN BIB024 for region proposals that can select salient regions of an image. Therefore, this method can attend to both object-level regions and other salient image regions. Park et al. BIB025 introduced a different type of attention-based image captioning method. This method can generate image captions addressing personal issues of an image. It mainly considers two tasks: hashtag prediction and post generation. This method uses a context sequence memory network (CSMN) to obtain the context information from the image. Descriptions of an image from a personalized view have a lot of applications in social media networks. For example, everyday people share a lot of images as posts on Facebook, Instagram, or other social media. Photo taking or uploading is a very easy task. However, describing the images is not easy because it requires a theme, sentiment, and context of the image. Therefore, the method considers past knowledge about the user's vocabularies or writing styles from the prior documents for generating image descriptions. In order to work with this new type of image captioning, the CSMN method has three contributions: First, the memory of this network can work as a repository and retain multiple types of context information. Second, the memory is designed in such a way that it can store all the previously generated words sequentially. As a result, it does not suffer from the vanishing gradient problem. Third, the proposed CNN can correlate with multiple memory slots, which is helpful for understanding contextual concepts. Attention-based methods have already shown good performance and efficiency in image captioning as well as other computer vision tasks. However, attention maps generated by these attention-based methods are only machine dependent. They do not consider any supervision from human attention. This creates the necessity to think about the gaze information and whether it can improve the performance of these attention methods in image captioning. Gaze indicates the cognition and perception of humans about a scene. Human gaze can identify the important locations of objects in an image. Thus, gaze mechanisms have already shown their potential performances in eye-based user modeling BIB002 BIB003 BIB010 BIB014 BIB015 , object localization  or recognition BIB005 , and holistic scene understanding BIB006 BIB007 . However, Sugano et al. BIB019 claimed that gaze information has not yet been integrated in image captioning methods. This method introduced human gaze with the attention mechanism of deep neural networks in generating image captions. The method incorporates human gaze information into an attention-based LSTM model BIB011 . For experiments, it uses the SALICON dataset BIB016 and achieves good results.