Summary: To train the MTM, the outputs of MTM are compared with the auto-encoded reference global textual representations via modality loss. During the inference, as illustrated in Fig. REF , the visual features are firstly extracted by the region detector such as Faster-RCNN. Secondly, before passing the visual features to the language decoder, Modality Transition Module (MTM) projects the averaged visual features into global textual representations. The language decoder decodes the representations into captions with the optional visual attention module such as the TopDown attention [1]}.
