In particular, this paper focuses on image captioning, which engages vision and language in a concise way.
Given an image, the vision encoder extracts the visual features from the RGB picture, then a language model decodes the visual representations into a sentence-level description, as illustrated in Fig.REF .
Typically, for the vision encoder, the CNN extracts visual feature maps to represent the visual content, followed by an optional region detector (e.g. Faster-RCNN) for recognising object-wise regions and outputting the refined regional features.
For the language decoder, the visual features initialise the language model as an input to generate the word one-by-one given the first “visual” word embeddings using Long Short-Term Memory (LSTM) [1]}.
Other techniques have been made to further improve the caption quality, such as incorporating visual attention, attributes, scene graphs, and reinforcement learning.
<FIGURE>