In image-grounded text generation, fine-grained representations of the image are considered to be of paramount importance. Most of the current systems incorporate visual features and textual concepts as a sketch of an image. However, plainly inferred representations are usually undesirable in that they are composed of separate components, the relations of which are elusive. In this work, we aim at representing an image with a set of integrated visual regions and corresponding textual concepts. To this end, we build the Mutual Iterative Attention (MIA) module, which integrates correlated visual features and textual concepts, respectively, by aligning the two modalities. We evaluate the proposed approach on the COCO dataset for image captioning. Extensive experiments show that the refined image representations boost the baseline models by up to 12 in terms of CIDEr, demonstrating that our method is effective and generalizes well to a wide range of models.
A number of neural approaches have been proposed to obtain image representations in various forms. An intuitive method is to extract visual features using a CNN or a RCNN. The former splits an image into a uniform grid of visual regions (Figure (a)), and the latter produces object-level visual features based on bounding boxes (Figure (b)), which has proven to be more effective. For image captioning, , and augmented the information source with textual concepts that are given by a predictor, which is trained to find the most frequent words in the captions. A most recent advance @cite_18 built graphs over the RCNN-detected visual regions, whose relationships are modeled as directed edges in a scene-graph, which is further encoded via a Graph Convolutional Network (GCN).
@cite_18
2890531016
It is always well believed that modeling relationships between objects would be helpful for representing and eventually describing an image. Nevertheless, there has not been evidence in support of the idea on image description generation. In this paper, we introduce a new design to explore the connections between objects for image captioning under the umbrella of attention-based encoder-decoder framework. Specifically, we present Graph Convolutional Networks plus Long Short-Term Memory (dubbed as GCN-LSTM) architecture that novelly integrates both semantic and spatial object relationships into image encoder. Technically, we build graphs over the detected objects in an image based on their spatial and semantic connections. The representations of each region proposed on objects are then refined by leveraging graph structure through GCN. With the learnt region-level features, our GCN-LSTM capitalizes on LSTM-based captioning framework with attention mechanism for sentence generation. Extensive experiments are conducted on COCO image captioning dataset, and superior results are reported when comparing to state-of-the-art approaches. More remarkably, GCN-LSTM increases CIDEr-D performance from 120.1 to 128.7 on COCO testing set.