Unlike TREC CAsT which is constructed semi-manually by IR experts, OR-QuAC is an open-retrieval conversational QA benchmark developed via crowdsourcing, as many other recently developed QA benchmarks [1]}, [2]}, [3]}.
OR-QuAC is designed for an open-retrieval setting where an IR system to be evaluated needs to retrieve passages from a collection, given a query and its conversational context [4]}.
OR-QuAC is an extension of the QuAC benchmark [2]}, which offers information-seeking conversations, to an open-retrieval setting by (1) creating a collection of over 11 million passages using the Wikipedia corpus that serves as the knowledge source of answering questions, and (2) generating relevance labels for query-passage pairs based on whether the passage contains the answer span of the query [6]}.
In addition, OR-QuAC aggregates the CANARD dataset [7]} which contains context-independent (de-contextualized) rewrites of QuAC queries.
