Abstract This paper considers the challenge of evaluating a set of classi ers, asdone in shared task evaluations like the KDD Cup or NIST TREC, without ex-pert labels. While expert labels provide the traditional cornerstone for evaluatingstatistical learners, limited or expensive access to experts represents a practicalbottleneck. Instead, we seek methodology for estimating performance of the clas-si ers (relative and absolute) which is more scalable than expert labeling yet pre-serves high correlation with evaluation based on expert labels. We consider both:1) using only labels automatically generated by the classi ers themselves (blindevaluation); and 2) using labels obtained via crowdsourcing. While crowdsourcingmethods are lauded for scalability, using such data for evaluation raises seriousconcerns given the prevalence of label noise. In regard to blind evaluation, twobroad strategies are investigated: combine & score and score & combine. Combine& Score methods infer a single -gold" label set by aggregating classi erlabels; classi ers are then evaluated based on this single pseudo-gold label set. Onthe other hand, score & combine methods: i) sample multiple label sets from clas-si er outputs, ii) evaluate classi ers on each label set, and iii) average classi erperformance across label sets. When additional crowd labels are also collected,we investigate two alternative avenues for exploiting them: 1) direct evaluation ofclassi ers; or 2) supervision of combine-and-score methods. To assess generality ofour techniques, classi er performance is measured using four common classi cationmetrics, with statistical signi cance tests establishing relative performance of theclassi ers for each metric. Finally, we measure both score and rank correlations be-tween estimated classi er performance vs. actual performance according to expertjudgments. Rigorous evaluation of classi ers from the TREC 2011 CrowdsourcingTrack shows reliable evaluation can be achieved without reliance on expert labels.
A broad concern with human labeling, such as relevance judging, is ensuring label consistency such that systems can be effectively trained and evaluated. A decade ago, Voorhees showed that reliable evaluation of search systems could be achieved despite significant variations in the underlying relevance judgments @cite_16 . @cite_3 took this idea further: could we forgo human labeling entirely (i.e., blind evaluation) by sampling relevant documents randomly after pooling outputs from all systems participating in a shared task evaluation? While such evaluation indeed correlated positively and significantly with use of human judgments, predicted performance was less reliable for the best systems. Aslam and Savell @cite_0 achieved comparable correlation by simply scoring each system by its mean Jaccard Coefficient over the set of retrieved documents vs. those retrieved by each other system. Wu and Crestani exploited similar reference count'' popularity, modeling expected correlation between relevance and the rank and frequency at which documents are retrieved by systems @cite_30 , similar to rank fusion @cite_27 . Several other blind evaluation methods have also been explored @cite_12 @cite_31 @cite_18 @cite_23 .
@cite_30 @cite_18 @cite_3 @cite_0 @cite_27 @cite_23 @cite_31 @cite_16 @cite_12
1983595289 1965862077 2783389297 2110930073
The most prevalent experimental methodology for comparing the effectiveness of information retrieval systems requires a test collection, composed of a set of documents, a set of query topics, and a set of relevance judgments indicating which documents are relevant to which topics. It is well known that relevance judgments are not infallible, but recent retrospective investigation into results from the Text REtrieval Conference (TREC) has shown that differences in human judgments of relevance do not affect the relative measured performance of retrieval systems. Based on this result, we propose and describe the initial results of a new evaluation methodology which replaces human relevance judgments with a randomly selected mapping of documents to topics which we refer to as pseudo-relevance judgments .Rankings of systems with our methodology correlate positively with official TREC rankings, although the performance of the top systems is not predicted well. The correlations are stable over a variety of pool depths and sampling techniques. With improvements, such a methodology could be useful in evaluating systems such as World-Wide Web search engines, where the set of documents changes too often to make traditional collection construction techniques practical. In batch evaluation of retrieval systems, performance is calculated based on predetermined relevance judgements applied to a list of documents returned by the system for a query. This evaluation paradigm, however, ignores the current standard operation of search systems which require the user to view summaries of documents prior to reading the documents themselves. In this paper we modify the popular IR metrics MAP and P@10 to incorporate the summary reading step of the search process, and study the effects on system rankings using TREC data. Based on a user study, we establish likely disagreements between relevance judgements of summaries and of documents, and use these values to seed simulations of summary relevance in the TREC data. Re-evaluating the runs submitted to the TREC Web Track, we find the average correlation between system rankings and the original TREC rankings is 0.8 (Kendall τ), which is lower than commonly accepted for system orderings to be considered equivalent. The system that has the highest MAP in TREC generally remains amongst the highest MAP systems when summaries are taken into account, but other systems become equivalent to the top ranked system depending on the simulated summary relevance. Given that system orderings alter when summaries are taken into account, the small amount of effort required to judge summaries in addition to documents (19 seconds vs 88 seconds on average in our data) should be undertaken when constructing test collections. To create a new IR test collection at minimal cost, we must carefully select which documents merit human relevance judgments. Shared task campaigns such as NIST TREC determine this by pooling search results from many participating systems (and often interactive runs as well), thereby identifying the most likely relevant documents in a given collection. While effective, it would be preferable to be able to build a new test collection without needing to run an entire shared task. Toward this end, we investigate multiple active learning (AL) strategies which, without reliance on system rankings: 1) select which documents human assessors should judge; and 2) automatically classify the relevance of remaining unjudged documents. Because scarcity of relevant documents tends to yield highly imbalanced training data for model estimation, we investigate sampling strategies to mitigate class imbalance. We report experiments on four TREC collections with varying scarcity of relevant documents, reporting labeling accuracy achieved, as well as rank correlation when evaluating participant systems using these labels vs. NIST judgments. Results demonstrate the effectiveness of our approach, coupled with further analysis showing how varying relevance scarcity, within and across collections, impacts findings. This paper addresses the problem of how to rank retrieval systems without the need for human relevance judgments, which are very resource intensive to obtain. Using TREC 3, 6, 7 and 8 data, it is shown how the overlap structure between the search results of multiple systems can be used to infer relative performance differences. In particular, the overlap structures for random groupings of five systems are computed, so that each system is selected an equal number of times. It is shown that the average percentage of a system's documents that are only found by it and no other systems is strongly and negatively correlated with its retrieval performance effectiveness, such as its mean average precision or precision at 1000. The presented method uses the degree of consensus or agreement a retrieval system can generate to infer its quality. This paper also addresses the question of how many documents in a ranked list need to be examined to be able to rank the systems. It is shown that the overlap structure of the top 50 documents can be used to rank the systems, often producing the best results. The presented method significantly improves upon previous attempts to rank retrieval systems without the need for human relevance judgments. This ''structure of overlap'' method can be of value to communities that need to identify the best experts or rank them, but do not have the resources to evaluate the experts' recommendations, since it does not require knowledge about the domain being searched or the information being requested.