Multi Scale Visual Semantics Aggregation with Self Attention for End to End Image Text Matching
For both datasets, input images are resized to 224 x 224 with a random crop, and we use the 152 layer ResNet He et al. (2016) pre trained on ImageNet Jia et al. (2009) to extract visual features. Different scales of feature maps (28 x 28, 14 x 14, 7 x 7) are captured by the last residual blocks in different stages (conv3_x, conv4_x, conv5_x) of ResNet. For the adaptive self attention (ASA) we follow Wang et al. (2018) to initialize the weight layers. Note that BatchNorm Ioffe and Szegedy (2015) is crucial to the convergence of ASA. We set the dimensionality of instance candidates and instance level features as C 2048 and D 1024 respectively. For sentences, the dimensionality of word embeddings is set as d 300, and the pre trained GloVe Jeffrey et al. (2014) is taken to initialize the word embedding matrix Wx. We set the number of hidden units in the bi directional GRU and the dimensionality of joint embedding space to 1024. We follow [Lee et al. (2018) Faghri et al. (2018) to set the margin m of triplet loss to 0.2 and threshold of maximum gradient norm to 2.0 for gradient clipping. The Adam optimizer Kingma and Ba (2015) is leveraged to train all models with a mini batch size of 128. Besides, two training stages are applied to each model, where we freeze ResNet in the first 20 epochs with an initial learning rate of 5e 4, and the whole network is fine tuned in the next 15 epochs with an initial learning rate of 2e 5. We follow Lee et al. (2018) to set the hyper parameters.
ACML