We study the problem of ranking from crowdsourced pairwise comparisons. Answers to pairwise tasks are known to be affected by the position of items on the screen, however, previous models for aggregation of pairwise comparisons do not focus on modeling such kind of biases. We introduce a new aggregation model factorBT for pairwise comparisons, which accounts for certain factors of pairwise tasks that are known to be irrelevant to the result of comparisons but may affect workers' answers due to perceptual reasons. By modeling biases that influence workers, factorBT is able to reduce the effect of biased pairwise comparisons on the resulted ranking. Our empirical studies on real-world data sets showed that factorBT produces more accurate ranking from crowdsourced pairwise comparisons than previously established models.
Paired comparison tests appear to be the most scientific form of product testing. This technique originated in the laboratory procedures of classical psychophysicists [7]. It closely approximates the laboratory experiment since the two test items are presented under controlled circumstances and the compared experiences are placed as close in time as possible. Like any controlled experiment, the paired comparison product test can be criticized for the artificiality of the circumstances under which the test items are used [1, 2]. This makes it imperative that paired product tests be carefully designed and that inferences about real-world behavior be made cautiously. When properly used, however, paired product tests are a powerful tool of the product researcher and continue to be one of the most popular approaches to preference measurement [3, 5, 6]. A phenomenon that has annoyed researchers who have used paired comparison tests is position bias or testing order effects. Many researchers have reported systematic biases in indicated preferences that can be related to the testing order of the two items. In some cases the bias favors the first item tested [4, 5]; in other cases it favors the second item [4]. An example of paired product test results with position bias was reported by Greenberg [5]. His results are shown in Table 1. A majority of participants who used Item A first indi ated a preference for Item A. Among those who tested Item B first, a majority reported a preference for B. A chi-square test indicated that the influence of order was significant at the .02 level [5]. When the two groups are combined, the effects of testing order are balanced, and there is no significant difference in indicated preferences across the sample. Obviously, misleading results would have occurred if all tests had been conducted with the same testing order. Because of the frequent occurrence of results such as these, it has become standard practice in conducting paired product tests to randomize the testing order.