In image-grounded text generation, fine-grained representations of the image are considered to be of paramount importance. Most of the current systems incorporate visual features and textual concepts as a sketch of an image. However, plainly inferred representations are usually undesirable in that they are composed of separate components, the relations of which are elusive. In this work, we aim at representing an image with a set of integrated visual regions and corresponding textual concepts. To this end, we build the Mutual Iterative Attention (MIA) module, which integrates correlated visual features and textual concepts, respectively, by aligning the two modalities. We evaluate the proposed approach on the COCO dataset for image captioning. Extensive experiments show that the refined image representations boost the baseline models by up to 12 in terms of CIDEr, demonstrating that our method is effective and generalizes well to a wide range of models.
In the field of image captioning, a prevailing paradigm is the encoder-decoder framework, where a CNN encoder and a RNN decoder are trained end-to-end, translating an image into a coherent description. To bridge the gap between the image and the half-finished caption, visual attention @cite_23 and semantic attention @cite_27 are separately proposed to force the decoder to focus on the most relevant visual regions and textual concepts, respectively, according to the generated context. As a result, the burden falls entirely on the decoder to associate the individual features, the relations of which are elusive. The contribution of this work is providing fine-grained image representations, which can be used in conjunction with the decoder-based attention mechanisms, and ultimately gives rise to higher-quality captions. It is worth noticing that used Transformer to replace RNN and showed that Transformer was less effective than RNN in image captioning, while we use the multi-head attention as a means for aligning the visual features and textual concepts, and the decoder still follows baselines and is not replaced.
@cite_27 @cite_23
2302086703 2950178297
Automatically generating a natural language description of an image has attracted interests recently both because of its importance in practical applications and because it connects two major artificial intelligence fields: computer vision and natural language processing. Existing approaches are either top-down, which start from a gist of an image and convert it into words, or bottom-up, which come up with words describing various aspects of an image and then combine them. In this paper, we propose a new algorithm that combines both approaches through a model of semantic attention. Our algorithm learns to selectively attend to semantic concept proposals and fuse them into hidden states and outputs of recurrent neural networks. The selection and fusion form a feedback connecting the top-down and bottom-up computation. We evaluate our algorithm on two public benchmarks: Microsoft COCO and Flickr30K. Experimental results show that our algorithm significantly outperforms the state-of-the-art approaches consistently across different evaluation metrics. Inspired by recent work in machine translation and object detection, we introduce an attention based model that automatically learns to describe the content of images. We describe how we can train this model in a deterministic manner using standard backpropagation techniques and stochastically by maximizing a variational lower bound. We also show through visualization how the model is able to automatically learn to fix its gaze on salient objects while generating the corresponding words in the output sequence. We validate the use of attention with state-of-the-art performance on three benchmark datasets: Flickr8k, Flickr30k and MS COCO.