Datasets  
We use the Flickr30k Entities [1]} and the MSCOCO [2]} image captioning datasets for evaluation.
They contain 31,783 images and 123,287 images, respectively. Each image in these two datasets is annotated with 5 sentences. In addition to textual captions, Flickr30k Entities [1]} contains 275,755 bounding boxes from 31,783 images and each bounding box is associated with the corresponding phrases in the caption.
MSCOCO does not have bounding boxes.
