Although answering questions on images is, arguably, more susceptible to automatic evaluation than the image description task [3, 8, 27] , ambiguities in the output space still remain. While such ambiguities can be handled using appropriate metrics [14, 15, 17, 26] , <cite>Visual Madlibs</cite> <cite>[32]</cite> has taken another direction, and handles them directly within the task. <cite>It</cite> asks machines to fill the blank prompted with a natural language description with a phrase chosen from four candidate completions (Figure 4 ).