These examples of system-oriented datasets, for document retrieval and result generation, have a common structure. They supply some content, a query, and some annotations indicating what content answers the query. It is then possible to evaluate two or more systems using the same fixed dataset, identifying which system had query responses that better match the ground truth annotation. The advantage of system-oriented evaluation is that it is possible to reuse the dataset across many different experiments and studies. One disadvantage is that it is not guaranteed that the dataset captures a valid real-world application. If the content, query or annotation fails to capture the requests and preferences of users on a real application, results on the dataset may tell us very little about the true quality of the systems being evaluated. Another disadvantage is that reuse of the dataset may lead to false conclusions via overfitting, where the same test data was used multiple times by the same researcher, leading the researcher to inadvertently make decisions using the test data. One way to avoid overfitting is to limit the number of iterations by having a leaderboard where each submission is publicly tracked, as in [1]}. Another is to have a single-shot evaluation where the test annotations are not even made public until after all submissions are finalized, as in [2]}. Another is to require that published studies evaluate the same approaches over multiple datasets without per-dataset iteration.
