A number of datasets on visual question answering have been introduced in recent years (Malinowski and Fritz 2014; Ren, Kiros, and Zemel 2015) , among which <cite>(Antol et al. 2015)</cite> in particular has gained the most attention and helped popularize the task. However, these datasets mostly consist of a small set of answers covering most of the questions, and most of the answers being single word.