Attention Supervision  
Recently, some works [1]}, [2]}, [3]} found that adding supervision to attention model is beneficial for captioning model.
More specifically, various approaches [1]}, [5]}, [3]}, [7]} have been proposed for various vision-and-language problems, e.g., referring expression [8]}, [8]} and grounding visual explanations [10]}.
However, for image captioning, these existing methods [1]}, [5]}, [3]} explicitly provide a series of ground truth bounding box annotations for training, while our model does not require such additional labeling.
Instead, our model learns the grounded attentional weights implicitly from the descriptions of images in a manner similar to self-supervision without bounding box annotations.
Therefore our approach can be directly applied to existing image captioning models.
