section1-8plus-4minus-46plus4minus4Experiments 
subsection2-8plus-4minus-46plus4minus4Experimental Settings
subsubsection3-4plus-4minus-4-1.5em plus-0.22em minus-0.1emDataset
The experiments are conducted on the MS-COCO [1]} Captioning dataset for evaluating the proposed MTM model. The train, validation, and test partitions are following the commonly adopted “Karpathy” [2]} split, which has 113,287 training, 5,000 validation, and 5,000 test images.
subsubsection3-4plus-4minus-4-1.5em plus-0.22em minus-0.1emImplementation Details
The proposed framework consists of the visual region detector, the MTM module, the text auto-encoder, the visual attention, and the language decoder.
For visual region detector, we extract the region features using Faster-RCNN with ResNet-101 backbone following [3]}.
In the MTM module, the model has two fully-connect neural network layers with 2048-1024 and 1024-1024 dimensions for each layer, i.e., \(d_v=2048\) , \(d_e=1024\) .
The text auto-encoder consists of an encoder and decoder, both of them are implemented with 2-layer LSTM with 1024 dimensions.
The captioning model is optimised by ADAM optimiser [4]}, and the learning rate is empirically set to \(5e-4\)  with decay.
All the models in this paper are trained on a server with GeForce GTX 1080Ti GPU.
