Our evaluation task shows human judges the full article, the anchor and answer sentences, and the generated question. We then ask them to judge the quality of the generated QUD
using a hierarchical schema shown in Figure REF .
Question 1 (Q1) assesses how reasonable the question is given context prior to and including the anchor sentence. The judges have four graded options: (1) yes for perfectly fine questions; (2) minor error for questions that contain minor typos or grammatical errors that do not impact its overall good quality; (3) sort-of for questions with non-negligible though also not catastrophic errors; and (4) no for questions that are not acceptable. (3) and (4) both contain subcategories representative for a sample of questions we closely inspected a priori.
Question 2 (Q2) assesses whether the question is answered by the targeted answer sentence, also with four graded options: (1) yes where the targeted answer sentence is clearly an answer to the generated question; (2) yes but not the main point where
the answer is not the at-issue content of the answer sentence. Such cases violate Grice's principle of quantity [1]} and QUD's principle that answers should be the at-issue content of the sentence [2]}. (3) Sort-of where the answer sentence is relevant to the question but it is questionable whether it actually addresses it; and (4) no where the generated question is clearly not addressed by the answer sentence. Annotators are allowed to skip Q2 if the generated question from Q1 is of lower quality.
