In previous research it has been shown that link-based web page metrics can be used to predict experts' assessment of quality. We are interested in a related question: do expert rankings of real-world entities correlate with search engine rankings of corresponding web resources? For example, each year US News & World Report publishes a list of (among others) top 50 graduate business schools. Does their expert ranking correlate with the search engine ranking of the URLs of those business schools? To answer this question we conducted 9 experiments using 8 expert rankings on a range of academic, athletic, financial and popular culture topics. We compared the expert rankings with the rankings in Google, Live Search (formerly MSN) and Yahoo (with list lengths of 10, 25, and 50). In 57 search engine vs. expert comparisons, only 1 strong and 4 moderate correlations were statistically significant. In 42 inter-search engine comparisons, only 2 strong and 4 moderate correlations were statistically significant. The correlations appeared to decrease with the size of the lists: the 3 strong correlations were for lists of 10, the 8 moderate correlations were for lists of 25, and no correlations were found for lists of 50.
Bharat and Mihaila @cite_15 propose a ranking scheme based on authority where the most authoritative pages get the highest ranking. Their algorithm is based on a special set of expert documents'' which are defined as web pages about a certain topic with many links to non-affiliated web pages on that topic. Non-affiliated pages are pages from different domains and with sufficiently different IP address. These expert documents are not chosen manually but automatically picked as long as they meet certain requirements (sufficient out-degree, etc). In response to a user query the most relevant expert documents are isolated. The proposed scheme locates relevant links within the expert documents and follows them to identify target pages. These pages are finally ranked according to the number and relevance of expert documents pointing to them and presented to the end user. Bharat and Mihaila evaluated their algorithm against three commercial search engines and found that it performs either just as good or in some cases even better than the top search engine when it comes to locating the home page of a specific topic. The same is true for discovering relevant pages to topic (where many good pages exist).
@cite_15
2088546029
Some methods for rank correlation in evaluation are examined and their relative advantages and disadvantages are discussed. In particular, it is suggested that different test statistics should be used for providing additional information about the experiments other that the one provided by statistical significance testing. Kendall's τ is often used for testing-rank correlation, yet it is little appropriate if the objective of the test is different from what τ was designed for. In particular, attention should be paid to the null hypothesis. Other measures for rank correlation are described. If one test statistic suggests to reject a hypothesis, other test statistics should be used to support or to revise the decision. The paper then focuses on rank correlation between webpage lists ordered by PageRank for applying the general reflections on these test statistics. An interpretation of PageRank behaviour is provided on the basis of the discussion of the test statistics for rank correlation.