We identify particular challenges that holistic tasks should exhibit and exemplify how they are manifested in <cite>a recent question answering challenge</cite> <cite>[27]</cite> . To judge competing architectures and measure the progress on the task, we suggest several directions to further improve existing metrics, and discuss different experimental scenarios.