Peer review, evaluation, and selection is a fundamental aspect of modern science. Funding bodies the world over employ experts to review and select the best proposals of those submitted for funding. The problem of peer selection, however, is much more general: a professional society may want to give a subset of its members awards based on the opinions of all members; an instructor for a MOOC or online course may want to crowdsource grading; or a marketing company may select ideas from group brainstorming sessions based on peer evaluation. We make three fundamental contributions to the study of procedures or mechanisms for peer selection, a specific type of group decision-making problem, studied in computer science, economics, and political science. First, we propose a novel mechanism that is strategyproof, i.e., agents cannot benefit by reporting insincere valuations. Second, we demonstrate the effectiveness of our mechanism by a comprehensive simulation-based comparison with a suite of mechanisms found in the literature. Finally, our mechanism employs a randomized rounding technique that is of independent interest, as it solves the apportionment problem that arises in various settings where discrete resources such as parliamentary representation slots need to be divided proportionally.
Peer review and peer selection are the cornerstone of modern science and hence, the quality, veracity, and accuracy of peer review and peer evaluation is a topic of interest across a broad set of disciplines. Most empirical evaluations of peer review and peer selection study the effectiveness and limits of the system, typically by assembling large corpora of peer-reviewed proposals and cross-examining them with new panels or review processes @cite_21 @cite_37 . Questions of bias, nepotism, sexism, cronyism, among other issues, have received extensive coverage, and have been substantiated to varying degrees, in the literature . However, a consistent conclusion in this research is that, in order to decrease the role of chance and or any systematic bias, the community needs to broaden the base of reviewers. Indeed, one way for the results of the review process to reflect the views of the entire scientific constituency and provide more value to the community is to increase the number of reviewers @cite_31 @cite_34 . The key scientific question lies in finding a mechanism that allows for crowdsourcing the work of reviewing, without compromising the incentives and quality of the peer review and selection process.
@cite_37 @cite_21 @cite_31 @cite_34
2278055525 1904238289 2033069678 2506952740
The National Science Foundation (NSF) will be exper- imenting with a new distributed approach to reviewing proposals, whereby a group of principal investigators (PIs) or proposers in a subfield act as reviewers for the proposals submitted by the same set of PIs. To encourage honesty, PIs chances for getting funded are tied to the quality of their reviews (with respect to the reviews provided by the entire group), in addition to the quality of their proposals. Intuitively, this approach can more fairly distribute the review workload, discourage frivolous proposal submission, and encourage high quality reviews. On the other hand, this method has already raised concerns about the integrity of the process and the possibility of strategic manipulation. In this paper, we take a closer look at three specific issues in an attempt to gai n a better understanding of the strengths and limitations of the new process beyond first impressions and anecdotal evidence . We start by considering the benefits and drawbacks of bundling the quality of PIs reviews with the scientific merit of their prop osals. We then consider the issue of collusion and favoritism. Finally, we examine whether the new process puts controversial proposals at a disadvantage. We conclude that some benefits of using revie w quality as an incentive mechanism may outweigh its drawbacks. On the other hand, even a coalition of two PIs can cause signifi cant harm to the process, as the built-in incentives are not strong enough to deter collusion. While we also confirm the common suspicion that the process is skewed toward non-controversial proposals, the more unexpected finding is that among equally controversial proposals, those of lower quality get a leg up through this process. Thus the process not only favors non-controversial proposals, but in some sense, mediocrity. We also discuss possible ways to improve this review process. When eliciting opinions from a group of experts, traditional devices used to promote honest reporting assume that there is an observable future outcome. In practice, however, this assumption is not always reasonable. In this paper, we propose a scoring method built on strictly proper scoring rules to induce honest reporting without assuming observable outcomes. Our method provides scores based on pairwise comparisons between the reports made by each pair of experts in the group. For ease of exposition, we introduce our scoring method by illustrating its application to the peer-review process. In order to do so, we start by modeling the peer-review process using a Bayesian model where the uncertainty regarding the quality of the manuscript is taken into account. Thereafter, we introduce our scoring method to evaluate the reported reviews. Under the assumptions that reviewers are Bayesian decision-makers and that they cannot influence the reviews of other reviewers, we show that risk-neutral reviewers strictly maximize their expected scores by honestly disclosing their reviews. We also show how the group's scores can be used to find a consensual review. Experimental results show that encouraging honest reporting through the proposed scoring method creates more accurate reviews than the traditional peer-review process. Formally published papers that have been through a traditional prepublication peer review process remain the most important means of communicating science today. Researchers depend on them to learn about the latest advances in their fields and to report their own findings. The intentions of traditional peer review are certainly noble: to ensure methodological integrity and to comment on potential significance of experimental studies through examination by a panel of objective, expert colleagues. In principle, this system enables science to move forward on the collective confidence of previously published work. Unfortunately, the traditional system has inspired methods of measuring impact that are suboptimal for their intended uses. Peer assessment is the most common approach to evaluating scientific work, and it is also gaining popularity for scaling evaluation of student work in large and distributed classes. The key idea is that each peer reviewer or grader rates a relatively small subset of the items, and that some method of manual, semi-automatic, or fully-automatic aggregation of all assessments defines the eventual rating of all items – the grade in peer grading, or whether to accept or reject a scientific manuscript. In this paper, we explore in how far a Bayesian Ordinal Peer Assessment (BOPA) method can provide additional decision support when making acceptance rejection decisions for a scientific conference. Using data from the 2015 ACM Conference on Knowledge Discovery and Data Mining (KDD), where this system was deployed, we discuss the potential merit of the BOPA approach compared to conventional decision support offered by the Microsoft Conference Management System (CMT).