Although related ideas have been explored for visual question answering [22] , and even have been used in <cite>Visual Madlibs</cite> <cite>[32]</cite> , we are first to show a significant improvement of such representation by using object proposals.