Rankings, representing preferences over a set of candidates, are widely used in many information systems, e.g., group decision making. It is of great importance to evaluate the consensus of the obtained rankings from multiple agents. There is often no ground truth available for a ranking task. An overall measure of the consensus degree enables us to have a clear cognition about the ranking data. Moreover, it could provide a quantitative indicator for consensus comparison between groups and further improvement of a ranking system. In this paper, a novel consensus quantifying approach, without the need for any correlation or distance functions, is proposed based on a concept of q-support patterns of rankings. The q-support patterns represent the commonality embedded in a set of rankings. A method for detecting outliers in a set of rankings is naturally derived from the proposed consensus quantifying approach. Experimental studies are conducted to demonstrate the effectiveness of the proposed approach.
Historically developed by Maurice Kendall in 1938 @cite_29 , Kendall's @math measures the correlation between two rankings by considering the numbers of pairwise items ranked in the same orders and in opposite orders. Suppose that we consider rankings over candidates @math . A ranking is an ordered list in which items in topper positions are more preferred than items in lower positions. Let @math be the position function. @math returns the position of item @math in ranking @math . The Kendall's @math for two rankings @math and @math is This coefficient is in the range @math , where value 1 correspons to the case that the two rankings are in the same order and value @math indicates that one ranking is in the reverse order of the other.
@cite_29
1985514943
1. In psychological work the problem of comparing two different rankings of the same set of individuals may be divided into two types. In the first type the individuals have a given order A which is objectively defined with reference to some quality, and a characteristic question is: if an observer ranks the individuals in an order B, does a comparison of B with A suggest that he possesses a reliable judgment of the quality, or, alternatively, is it probable that B could have arisen by chance? In the second type no objective order is given. Two observers consider the individuals and rank them in orders A and B. The question now is, are these orders sufficiently alike to indicate similarity of taste in the observers, or, on the other hand, are A and B incompatible within assigned limits of probability? An example of the first type occurs in the familiar experiments wherein an observer has to arrange a known set of weights in ascending order of weight; the second type would arise if two observers had to rank a set of musical compositions in order of preference. The measure of rank correlation proposed in this paper is capable of being applied to both problems, which are, in fact, formally very much the same. For purposes of simplicity in the exposition it has, however, been thought convenient to preserve a distinction between theni.