The major issue of OR-QuAC is data quality. The questions written by crowd workers are of lower quality than the TREC CAsT queries written by IR experts. This is expected because a crowd worker is unlikely to spend as much time as an IR expert in making interesting and challenging information seeking conversations for evaluating IR systems.
Another issue is that the interactive information-seeking task designed for collecting conversational QA pairs is artificial and does not reflect the complexity of real-world conversational search scenarios. For example, it has been reported that the dialog trajectories in QuAC often follow a pattern that students start a dialog by asking a question about the beginning of the section before progressing to asking questions about the end [1]}.
The setting also ensures that the answer span is in the same document section based on which the question is generated.
These artifacts can be easily exploited by machine-learning models to produce good results only on the OR-QuAC benchmark, which are not generalizable to real-world settings.
In addition, the automatic relevance labeling process is problematic. The process labels a passage as relevant to a query based on whether it contains the answer of the query. However, the passage that contains the answer span might not be truly relevant if it does not include enough evidence to answer the question. On the other hand, there could be many relevant passages that do not contain the answer span.
<TABLE><TABLE>