Experimental setting: We follow the setting by [1]}, where bottom-up features extracted from images by Faster R-CNN [2]} are used as visual features, pretrained word-embeddings [3]} and LSTM [4]} are used to extract question features. We adopt the encoder-decoder structure in MCAN with six co-attention layers. We use the same model hyperparameters and training settings in [1]} as follows:
the dimensionality of input image features, input question features, and fused multi-modal features are set to be 2048, 512, and 1024, respectively. The latent dimensionality in the multi-head attention is 512, the number of heads is set to 8, and the latent dimensionality for each head is 64. The size of the answer vocabulary is set to \(N = 3129\)  using the strategy in [6]}. To train the MCAN model, we use the Adam optimizer [7]} with \(\beta _1 = 0.9\)  and \(\beta _2 = 0.98\) . The base learning rate is set to \(\min (2.5te^{-5}, 1e^{-4})\) , where \(t\)  is the current epoch number starting from 1. After 10 epochs, the learning rate is decayed by \(1/5\)  every 2 epochs. All the models are trained up to 13 epochs with the same batch size of 64.
