This section describes in detail two widely used benchmarks of conversational search, the TREC CAsT benchmark [1]}, [2]} and the OR-QuAC dataset [3]}.
They are representative examples of two benchmark construction approaches, TREC and crowdsourcing, respectively.
TREC CAsT is a standard TREC-style benchmark which is constructed by the IR community to evaluate IR systems, following the test collection evaluation paradigm [4]}, [5]}. “At the core of this methodology was the idea that live users could be removed from the evaluation loop, thus simplifying the evaluation and allowing researchers to run in vitro–style experiments in a laboratory with just their retrieval engine, a set of queries, a test collection, and a set of judgments (i.e., a list of relevant documents).”
OR-QuAC is an open-retrieval conversational question answering (QA) benchmark constructed via crowdsourcing.
It is initially curated using crowdsourcing for conversational QA and then extended for the conversational search task by deriving document-level relevance labels from the answer annotations [6]}.
We end this section by briefly reviewing other benchmarks.
