A Comprehensive Survey of Deep Learning for Image Captioning <s> Novel-Object-Based Image Captioning. <s> While recent deep neural network models have achieved promising results on the image captioning task, they rely largely on the availability of corpora with paired image and sentence captions to describe objects in context. In this work, we propose the Deep Compositional Captioner (DCC) to address the task of generating descriptions of novel objects which are not present in paired imagesentence datasets. Our method achieves this by leveraging large object recognition datasets and external text corpora and by transferring knowledge between semantically similar concepts. Current deep caption models can only describe objects contained in paired image-sentence corpora, despite the fact that they are pre-trained with large object recognition datasets, namely ImageNet. In contrast, our model can compose sentences that describe novel objects and their interactions with other objects. We demonstrate our model's ability to describe novel concepts by empirically evaluating its performance on MSCOCO and show qualitative results on ImageNet images of objects for which no paired image-sentence data exist. Further, we extend our approach to generate descriptions of objects in video clips. Our results show that DCC has distinct advantages over existing image and video captioning approaches for generating descriptions of new objects in context. <s> BIB001 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Novel-Object-Based Image Captioning. <s> Image captioning often requires a large set of training image-sentence pairs. In practice, however, acquiring sufficient training pairs is always expensive, making the recent captioning models limited in their ability to describe objects outside of training corpora (i.e., novel objects). In this paper, we present Long Short-Term Memory with Copying Mechanism (LSTM-C) --- a new architecture that incorporates copying into the Convolutional Neural Networks (CNN) plus Recurrent Neural Networks (RNN) image captioning framework, for describing novel objects in captions. Specifically, freely available object recognition datasets are leveraged to develop classifiers for novel objects. Our LSTM-C then nicely integrates the standard word-by-word sentence generation by a decoder RNN with copying mechanism which may instead select words from novel objects at proper places in the output sentence. Extensive experiments are conducted on both MSCOCO image captioning and ImageNet datasets, demonstrating the ability of our proposed LSTM-C architecture to describe novel objects. Furthermore, superior results are reported when compared to state-of-the-art deep models. <s> BIB002 </s> A Comprehensive Survey of Deep Learning for Image Captioning <s> Novel-Object-Based Image Captioning. <s> Recent captioning models are limited in their ability to scale and describe concepts unseen in paired image-text corpora. We propose the Novel Object Captioner (NOC), a deep visual semantic captioning model that can describe a large number of object categories not present in existing image-caption datasets. Our model takes advantage of external sources -- labeled images from object recognition datasets, and semantic knowledge extracted from unannotated text. We propose minimizing a joint objective which can learn from these diverse data sources and leverage distributional semantic embeddings, enabling the model to generalize and describe novel objects outside of image-caption datasets. We demonstrate that our model exploits semantic information to generate captions for hundreds of object categories in the ImageNet object recognition dataset that are not observed in MSCOCO image-caption training data, as well as many categories that are observed very rarely. Both automatic evaluations and human judgements show that our model considerably outperforms prior work in being able to describe many more categories of objects. <s> BIB003
Despite recent deep-learning-based image captioning methods having achieved promising results, they largely depend on the paired image and sentence caption datasets. These types of methods can only generate a description of the objects within the context. Therefore, the methods require a large set of training image-sentence pairs. Novel-object-based image captioning methods can generate descriptions of novel objects that are not present in paired image-caption datasets. The methods of this category have the following general steps: (1) A separate lexical classifier and a language model are trained on unpaired image data and unpaired text data. A simple block diagram of a novel-object-based image captioning method is given in Figure 9 . Current image captioning methods are trained on image-caption paired datasets. As a result, if there are unseen objects in the test images, they cannot present them in their generated captions. Anne et al. BIB001 proposed a deep compositional captioner (DCC) that can represent the unseen objects in generated captions. Yao et al. BIB002 proposed a copying mechanism to generate a description for novel objects. This method uses a separate object recognition dataset to develop classifiers for novel objects. It integrates the appropriate words in the output captions by a decoder RNN with a copying mechanism. The architecture of the method adds a new network to recognize the unseen objects from unpaired images and incorporate them with the LSTM to generate captions. Generating captions for the unseen images is a challenging research problem. Venugopalan et al. BIB003 introduced a novel object captioner (NOC) for generating captions for unseen objects in the image. They used external sources for recognizing unseen objects and learning semantic knowledge.