Settings  
In implementation, we use spaCy library [1]} for noun phrase tagging. We set the \(\lambda =0.1\) , according to the average performance on the validation set.
We experiment on three representative models Up-Down [2]}, GVD [3]} and AoANet [4]}.
Since our approach aims to force the conventional attention model can learn to ground each output word to image regions and is augmentative to the existing models, we keep the inner structure of the baselines untouched and preserve the original settings.
Our code is implemented in PyTorch [5]}. All re-implementations and our experiments were ran on V100 GPUs.
Following common practice [2]}, [4]}, [8]}, [9]}, [10]}, we further adopt CIDEr-based training objective using reinforcement training [11]}.
In particular, inspired by the [12]}, we further introduce a simple Prophet Knowledge Distillation (PKD) trick to distill the future knowledge for image captioning. The introduced PKD trick can be used to boost the performance during the reinforcement training.
To conduct a fair comparison, both the baseline models and our method have been equipped with the PKD trick.
Due to limited space, for detailed description of the PKD trick and the settings, please refer to our supplementary materials.
