Focused Evaluation for Image Description with Binary Forced Choice Tasks
Current evaluation metrics for image description may be too coarse. We therefore propose a series of binary forced choice tasks that each focus on a different aspect of the captions. We evaluate a number of different off the shelf image description systems. Our results indicate strengths and shortcomings of both generation and ranking based approaches.
VL@ACL