For our experiments, we use the top-down single-sentence captioning model in Anderson et al. (2017) . This model is similar to the "flat" model in <cite>Krause et al. (2016)</cite> , except that it incorporates attention with a top-down mechanism.