The previous section described a typical non-conversational dataset as having three main aspects: The query, the content and the annotation of which content is correct for the query. This section describes how the query may be interpreted and annotated in a conversational context. The main difference is that we would now describe a dataset as having: query+context, content and annotation. The annotation, and potentially the content selection, can now take the context into account. Table REF  summarizes a number of benchmarks and other datasets in this way. The non-conversational IR test collection approach is exemplified by the MS MARCO passage dataset [1]}. The corpus contains 8.8 million passages, the queries do not have any conversational context, and the annotations are passage-level relevance judgments with respect to a query. The answer extraction approach we described is SQuAD [2]}, which has no special conversational context and is evaluated based on identifying correct spans from a paragraph. We also described TREC CaST, which is related to MS MARCO, but adds the conversational context. This makes it much more suitable for CIR evaluation, since we have argued that user revealment over multiple steps of conversation is an important property of a CIR process.
<TABLE>