In QUD annotation and DCQA itself [1]}, [2]}, [3]}, it is often the case that multiple questions can be asked even given the same anchor and/or answer sentences.
The evaluation of QUD validity thus involves complex reasoning performed jointly among (long) context, the anchor, the answer sentence, and the generated question itself. For these reasons, we rely on human evaluation, and leave the development of automatic evaluation metrics for future work.Existing automatic measures for open-ended tasks are known to correlate poorly with human judgments [4]}, [5]}; additionally, whether the answer sentence actually answers the question is a key aspect to validate QUD. But as shown in [3]}, poor performance bars existing QA models from being used to evaluate QUD parsing. Appendix  discusses anchor prediction “accuracies” against human-annotated anchors in DCQA.
