Datasets. We experiment on the MSCOCOhttp://cocodataset.org/ dataset, which is the most popular dataset for image captioning. The original dataset contains about 82,783 training images and 40,504 validation images. Following most of the previous work, we first evaluate our model on the “Karpathy” data split [1]} with 5,000 images for validation, 5,000 images for testing, and the rest for training. Then we evaluate our model on the on-line COCO test server. We set the maximum decoding length to 16 which equals the maximum length of \(95\%\)  of captions. We keep the words that occur no less than 5 times, resulting in a vocabulary of 10,369 words.
