System-oriented evaluation of a result generation component can be thought of as evaluation of a machine reading comprehension system (Chapter ), to see how well the system can generate or extract an answer from a retrieved paragraph of text. Take extractive readers (Section REF ) as an example. Each instance in the evaluation considers a particular question and paragraph, capturing the user's required answer by identifying one or more spans of paragraph text that answer the question [1]}. Identifying multiple spans is useful to identify the human level of agreement on the task, and to capture that there can be multiple possible correct spans.
