Scientific research funding is allocated largely through a system of soliciting and ranking competitive grant proposals. In these competitions, the proposals themselves are not the deliverables that the funder seeks, but instead are used by the funder to screen for the most promising research ideas. Consequently, some of the funding program' s impact on science is squandered because applying researchers must spend time writing proposals instead of doing science. To what extent does the community' s aggregate investment in proposal preparation negate the scientific impact of the funding program? Are there alternative mechanisms for awarding funds that advance science more efficiently? We use the economic theory of contests to analyze how efficiently grant proposal competitions advance science, and compare them with recently proposed, partially randomized alternatives such as lotteries. We find that the effort researchers waste in writing proposals may be comparable to the total scientific value of the research that the funding supports, especially when only a few proposals can be funded. Moreover, when professional pressures motivate investigators to seek funding for reasons that extend beyond the value of the proposed science (e. g. , promotion, prestige), the entire program can actually hamper scientific progress when the number of awards is small. We suggest that lost efficiency may be restored either by partial lotteries for funding or by funding researchers based on past scientific success instead of proposals for future work.
Over the past 50 years, research funding in the United States has failed to keep pace with growth in scientific activity. Funding rates in grant competitions have plummeted (S1 Fig, [1–4]) and researchers spend far more time writing grant proposals than they did in the past [5]. A large survey of top US universities found that, on average, faculty devote 8% of their total time—and 19% of their time available for research activities—towards preparing grant proposals [6]. Anecdotally, medical school faculty may spend fully half their time or more seeking grant funding [5,7]. While the act of writing a proposal may have some intrinsic scientific value [8]—perhaps by helping an investigator sharpen ideas—much of the effort given to writing proposals is effort taken away from doing science [9]. With respect to scientific progress, this time is wasted [10]. Frustrated with the inefficiencies of the current funding system, some researchers have called for an overhaul of the prevailing funding model [9,11–18]. In particular, Fang and Casadevall [16] recently suggested a partial lottery, in which proposals are rated as worthy of funding or not, and then a subset of the worthy proposals are randomly selected to receive funds. Arguments in favor of a partial lottery include reduced demographic and systemic bias, increased transparency, and a hedge against the impossibility of forecasting how scientific projects will unfold [16]. Indeed, at least three funding organizations—New Zealand' s Health Research Council and their Science for Technological Innovation program, as well as the Volkswagen Foundation [19]—have recently begun using partial lotteries to fund riskier, more exploratory science. Compared with a proposal competition, a lottery permits more proposals to qualify for funding and thus lowers the bar that applicants must clear. A lottery also offers a lower reward for success, as a successful proposal receives a chance at funding, not a guarantee of funding. Thus, we expect that investigators applying to a partial lottery will invest less time and fewer resources in writing a proposal. To a first approximation, then, a proposal competition funds high-value projects while wasting substantial researcher time on proposal preparation, whereas a partial lottery would fund lower-value projects on average but would reduce the time wasted writing proposals. It is not obvious which system will have the greater net benefit for scientific progress. In this article, we study the merits and costs of traditional proposal competitions versus partial lotteries by situating both within the rich economic theory of contests. In this theory, competing participants make costly investments (" bids" ) in order to win one or more prizes [20,21]. Participants differ in key attributes, such as ability and opportunity cost, that determine their optimal strategies. In an economics context, contests are often used by the organizer as a mechanism to elicit effort from the participants. For example, TopCoder and Kaggle are popular contest platforms for tech firms (the organizers) to solicit programming or data-analysis effort from freelance workers (the participants). However, because the participants' attributes influence their optimal strategies, the bids that participants submit reveal those attributes. Thus, screening of participants often arises as a side effect. In funding competitions, the organizer is a funding body and the participants are competing investigators. Investigators pitch project ideas of varying scientific value by preparing costly proposals. However, unlike a traditional economic contest, the funding body' s primary objective is to identify the most promising science, using proposals to screen for high-value ideas. The funding body has little interest in eliciting work during the competition itself, as the proposals are not the deliverables the funder seeks. All else equal, the funding agency would prefer to minimize the work that goes into preparing proposals, to leave as much time as possible for investigators to do science. In this case, how should the funder organize the contest to support promising science without squandering much of the program' s benefit on time wasted writing proposals? Below, we pursue this question by presenting and analyzing a contest model for scientific funding competitions. We first use the model to assess the efficiency of proposal competitions for promoting scientific progress and ask how that efficiency depends on how many proposals are funded. We next explore how efficiency is impacted when extrascientific incentives such as professional advancement motivate scientists to pursue funding, and compare the efficiency of proposal competitions with that of partial lotteries. Finally, we reflect on alternative ways to improve the efficiency of funding competitions without adding intentional randomness to the award process. All of our analyses focus on equilibrium behavior and thus pertain most directly to long-standing funding competitions, for which researchers can acquire experience that informs their future actions.
We use the model to explore how efficiently the grant competition advances science. From the perspective of an individual investigator, the investigator' s return on her investment (ROI) is the ratio of her payoff to the cost of her bid: Investigator′sROI= (v0+v) η (b (v) ) − (1−k) c (v, b (v) ) c (v, b (v) ). (3) An investigator will never choose to write a proposal that generates a negative payoff, because she can always obtain a payoff of 0 by opting out. (If the investigator opts out, Eq 3 evaluates to 0/0, in which case we define her ROI to be 0.) Thus, an investigator' s equilibrium ROI must be ≥0. To analyze the funding program' s impact on scientific progress as a whole, we compare the total value of the science that the funding program supports with the total value of the science that has been squandered preparing proposals. Of course, both of these quantities will be confounded with the number of grants that are funded, so we standardize to a per-funded-proposal basis. In notation, the average scientific value per funded proposal is 1p∫vη (b (v) ) dF (v), (4) and the average scientific waste per funded proposal is 1p∫ (1−k) c (v, b (v) ) dF (v). (5) We will refer to the difference between these two quantities as the scientific gain (or loss, should it be negative) per funded proposal, which is our measure of the funding program' s scientific efficiency. Note that while an investigator will never enter a grant competition against her own self interest, there is no guarantee that the scientific value per funded proposal will exceed the scientific waste. This is because the investigator' s payoff includes private, extrascientific rewards obtained by winning a grant (v0), and (in our accounting, at least) these extrascientific rewards do not benefit the funding agency. If extrascientific motivations for winning grants are large enough, investigators may enter a grant competition even when doing so decreases their scientific productivity. If enough investigators are motivated accordingly, then the scientific progress sacrificed to writing proposals could exceed the scientific value of the funding program. In this case, the grant competition would operate at a loss to science, and the funding agency could do more for science by eschewing the proposal competition and spreading the money evenly among active researchers in the field, or by giving the money to researchers selected entirely at random.
We illustrate the model' s behavior by choosing a few possible sets of parameter values. Our parameter choices are not directly informed by data. Thus, while the numerical examples illustrate the model' s possible behavior, we highlight the results that are guaranteed to hold in general. Throughout, we use the following baseline set of parameters. We assume that the project values, v, have a triangular distribution ranging from vmin = 0. 25 to vmax = 1 with a mode at vmin, such that low-value ideas are common and high-value ideas are rare (i. e. , F (v) = 1− (16/9) (1−v) 2). For the cost function, we choose c (v, x) = x2/v. We choose a convex dependence on x to suggest that the marginal cost of improving a proposal increases as the proposal becomes stronger. We assume that the intrinsic scientific value of writing a proposal allows investigators to recoup k = 1/3 of the disutility cost of proposal preparation. We first explore the case when investigators are motivated purely by the scientific value of their projects (v0 = 0) and then introduce extrascientific benefits (v0 = 0. 25). In S1 Text and S3–S6 Figs, we provide parallel results with two alternative parameter sets. The evaluation process by which review panels rank proposals introduces a layer of randomness to the awarding of grants [23–25]. To capture noisy assessment, we use a bivariate copula [26] to specify the joint distribution of a proposal' s actual quantile, and its quantile as assessed by the funding agency' s review panel. A bivariate copula is a probability distribution on the unit square that has uniformly distributed marginals, as all quantiles must. We use a Clayton copula [27], which allows for accurate assessment of weak proposals, but noisier assessment of strong proposals (S2 Fig). This choice is motivated by the pervasive notion that review panels can readily distinguish strong proposals from weak ones, but struggle to discriminate among strong proposals [16,25,28]. A Clayton copula has a single parameter (θ) that controls how tightly its two components are correlated. Rather arbitrarily, we use θ = 10 in the baseline parameter set. The Clayton copula has the important property that a proposal' s probability of funding increases monotonically as its strength increases, regardless of the payline. Thus, we exclude the possibility that panels systematically favor weaker proposals. By using a copula, we implicitly assume that η (x) depends on x only through its rank. In S1 Text, we show how a copula leads to an equation for ξ′ (v), which can then be plugged in to Eq 2. Fig 2 shows numerical results for the baseline parameters at generous (p = 45%) and low (p = 15%) paylines. In this particular case, investigators' payoffs fall faster than costs as paylines drop, leading to a reduced ROI for everyone at the lower payline (Fig 2B). We will argue below that every investigator' s ROI must inevitably fall when the payline becomes small (see S2 and S3 Figs for additional examples). From the funding agency' s perspective, with our baseline parameters, both the average scientific value and average waste per funded proposal increase as the payline falls, for paylines below 50% (Fig 3A). However, as the payline decreases, waste escalates more quickly than scientific value, reducing the scientific gain per funded project (Fig 3B). This same result also appears in our alternative parameter sets (S5 and S6 Figs). We will argue below that the decline in scientific efficiency at low paylines is an inevitable if unfortunate characteristic of proposal competitions. Clearly, quantitative details of the model' s predictions depend on the parameter inputs. To understand the robustness of these predictions, it helps to study the case in which panels discriminate perfectly among proposals. While perfect discrimination is obviously unrealistic in practice, it yields a powerful and general set of results that illuminate how the model behaves when discrimination is imperfect. Numerical results for perfect discrimination under the baseline parameter set appear in S7 and S8 Figs. At equilibrium under perfect assessment, every project above a threshold value v* = F−1 (1−p) will receive funding, and no project idea below this threshold will be funded. Investigators with projects of value v>v* all prepare proposals to the identical strength x* = h−1[ (v0+v*) / ( (1−k) g (v*) ) ] and are funded with certainty. Investigators with projects of value v<v* opt out (S7 Fig). All of the subsequent results follow (details appear in S1 Text). First, as paylines drop, all investigators realize either a diminishing or zero ROI, because investigators who remain in the competition must pay a higher cost for a reduced payoff. Second, the average scientific value per funded proposal must increase as paylines drop, because only the highest-value projects are funded under low paylines. Third, in the limiting case in which only one of many proposals can be funded (technically, the limit as p approaches 0 from above), the scientific value and scientific waste associated with the last funded project converge, and science is no better off than if no grant had been given at all (S8 Fig). With perfect assessment, there is no general relationship between the scientific efficiency of a proposal competition and the payline that holds across the full range of paylines (but see Hoppe and colleagues [22] for a sharp result when the cost function is independent of v). Of course, we wouldn' t expect scientific efficiency to decline monotonically with a falling payline, because there are likely to be some low-value projects that can be weeded out at low cost. However, our last result above guarantees that the scientific gain per funded proposal must eventually vanish as the payline declines to a single award. Returning to the reality of imperfect discrimination, as long as review panels do not systematically favor weaker proposals, noisy assessment changes little about these qualitative results. That is, investigators' ROIs will drop as paylines fall, the average scientific value per funded proposal will increase as paylines decrease, and the scientific efficiency of the proposal competition must eventually decline as the payline approaches a single award. But efficiency need not drop to zero. Perhaps counterintuitively, imperfect discrimination is a saving grace at low paylines. Noisy assessment discourages top investigators from pouring excessive effort into grant writing as paylines fall, because the marginal benefit of writing an even better grant becomes small when review panels struggle to discriminate among top proposals. Indeed, noisy assessment, unlike perfect discrimination, allows a proposal competition to retain a positive impact on science, even with a single funded grant (compare Fig 3 and S8 Fig). This result hints at the salutary nature of randomness at low paylines, which we will see more vividly when we consider lotteries below. Thus far, we have considered the case in which investigators are motivated only by the scientific value of the projects proposed (v0 = 0). Now, suppose that investigators are additionally motivated by the extrascientific benefits of receiving a grant, such as professional advancement or prestige (v0>0). Eq 2 shows that adding extrascientific motivation will increase the effort that investigators devote to preparing grant proposals. However, in our model, at least, this extra effort has no bearing on which grants are funded and thus does not affect the scientific value of the grants that are awarded. Increasing scientific costs without increasing scientific value will clearly be detrimental to the funding program' s scientific efficiency. Extrascientific benefits to investigators can even cause the entire funding program to operate at a loss to science when paylines are low (Fig 3).
Our model can also be used to analyze the efficiency of a partial lottery for advancing science. Suppose that a fraction q≥p of proposals qualify for the lottery, and each qualifying proposal is equally likely to be chosen for funding. Call q the" lottery line." Now, the investigator' s payoff is (p/q) (v0+v) ηl (x) − (1−k) c (v, x), where ηl (x) is the equilibrium probability that the proposal qualifies for the lottery. In S1 Text, we show that the investigator' s bid is given by b (v) =h−1[pq11−k∫0vv0+tg (t) ξl′ (t) dt] (6) where ξl (v) = ηl (b (v) ). Our major result for lotteries is that measures of scientific efficiency—expressions 3,4, and 5—depend on the lottery line q but are independent of the payline p (proofs appear in S1 Text). This result follows from the fact that, in a lottery, each investigator' s benefit and cost are proportional to p. Thus, an investigator' s ROI and the scientific efficiency of the funding program are determined by the lottery line but are not affected by the payline. To illustrate, Fig 4 compares an investigator' s costs and benefits in a proposal competition with 45%, 30%, and 15% paylines versus a partial lottery with a q = 45% lottery line and the same three paylines. The key feature of Fig 4 is that the investigator' s benefit curve in a partial lottery scales in such a way that her ROI is the same for any payline ≤q. Consequently, a partial lottery with a lottery line of q and any payline ≤q achieves the same scientific efficiency as a proposal competition with a payline of q. Thus, our numerical results showing the investigator' s ROI (Fig 2B) or the scientific efficiency (Fig 3) in a funding competition also show the efficiency of a lottery with the equivalent lottery line. That is, a lottery in which 45% of applicants qualify for the lottery has the same scientific efficiency as a proposal competition with a 45% payline, regardless of how many proposals in the lottery are randomly selected for funding. Thus, a lottery can restore the losses in efficiency that a proposal competition suffers as paylines become small. In S1 Text, we also analyze a more general type of lottery in which proposals are placed into one of a small number of tiers, with proposals in more selective tiers awarded a greater chance of funding [13,17,29]. In a multitiered lottery, the efficiency is entirely determined by the number of tiers and the relative probabilities of funding in each, and is independent of the payline. Numerical results (S9 Fig) illustrate that the scientific value and waste of a multitiered lottery fall in between those of a proposal competition and a single-tiered lottery. Thus, a multitiered lottery offers an intermediate design that would partially reduce the waste associated with preparing proposals, while still allowing review panels to reward the best proposals with a higher probability of funding.
Our major result is that proposal competitions are inevitably and inescapably inefficient mechanisms for funding science when the number of awards is smaller than the number of meritorious proposals. The contest model presented here suggests that a partially randomized scheme for allocating funds—that is, a lottery—can restore the efficiency lost as paylines fall, albeit at the expense of reducing the average scientific value of the projects that are funded. Why does a lottery disengage efficiency from the payline, while a proposal competition does not? For investigators, proposal competitions are, to a first approximation, all-or-nothing affairs, because an investigator only obtains a substantial payoff if her grant is funded. At high paylines (or, more precisely, when the number of awards matches the number of high-value projects), investigators with high-value projects can write proposals that win funding at modest cost to themselves. As the number of awards dwindles, however, competition stiffens. Depending on the details of the assessment process, an investigator with a high-value project must either work harder for the same chance of funding, or work just as hard for a smaller chance of funding. Either way, the return on her investment declines sharply. Thus, a contest is most efficient at the payline that weeds out low-value projects but does not attempt to discriminate among the high-value projects (e. g. , S6 Fig). At lower paylines, however, the effort needed to signal which projects are most valuable begins to approach the value of those projects, making the funding program less worthwhile. In a lottery, investigators do not compete for awards per se, but instead compete for admission to the lottery. The value to the investigator of being admitted to the lottery scales directly with the number of awards. It turns out that both the investigator' s expected benefit and her costs of participation scale directly with the payline and thus the payline has no effect on efficiency. (In S1 Text, we follow Hoppe and colleagues [22] to show that this scaling can be explained by the economic principle of revenue equivalence.) If there are fewer awards than high-value projects, a lottery that weeds out the low-value projects but does not attempt to discriminate among high-value projects will facilitate scientific progress more efficiently than a contest. Unfortunately, empirical comparisons between the efficiencies of funding competitions versus partial lotteries do not yet exist, to the best of our knowledge. However, two recent anecdotes support our prediction that the waste in proposal competitions is driven by the strategic dynamics of the contest itself. First, in 2012, the US National Science Foundation' s Divisions of Environmental Biology and Integrative Organismal Systems switched from a twice-per-year, one-stage proposal competition to a once-per-year, two-stage competition, in part to reduce applicants' workload. However, the switch failed to reduce the applicants' aggregate workload meaningfully [30], and the two-stage mechanism was subsequently abandoned. Second, in 2014, the National Health and Medical Research Council of Australia streamlined the process of applying for their Project Grants, cutting the length of an application in half [31]. However, researchers spent more time, not less, preparing proposals after the process had been streamlined, both individually and in aggregate [31]. Both of these experiences are consistent with our prediction that, in a proposal competition, the effort that applicants expend is dictated by the value of funding to the applicants and the number of awards available, but does not depend on the particular format of the proposals. A lottery is a radical alternative, and may be politically untenable [32]. If a lottery is not viable, an alternative approach to restoring efficiency is to design a contest in which the effort given to competing for awards has more direct scientific value. For example, a contest that rewards good science in its completed form—as opposed to rewarding well-crafted proposals that describe future science—motivates the actual practice of good science, and will be less wasteful at low paylines [9,14]. Program officers could also be given the discretion to allocate some funds by proactively scouting for promising researchers or projects. Of course, a contest based on completed science or scouting has its own drawbacks, including rich-getting-richer feedback loops, a risk of new barriers to entry for investigators from historically underrepresented demographic groups, and the Goodhart' s law phenomenon, whereby a metric that becomes a target ceases to be a good metric [33]. Nevertheless, it is tantalizing to envision a world in which the resources that universities currently devote to helping researchers write proposals are instead devoted to helping researchers do science. This analysis also shows that extrascientific professional incentives to pursue grant funding can damage the scientific efficiency of a proposal competition. As many of these extrascientific incentives arise from administrators using grant success as a primary yardstick of professional achievement, perhaps one major benefit of adding explicit randomness to the funding mechanism would be to compel administrators to de-emphasize grant success in professional evaluations. Alternatively, to the degree that administrators value and reward grant success because of the associated overhead funds that flow to the university, funding agencies could reduce waste by distributing overhead separately from funding awards. Instead, perhaps overhead could be allocated based partially on the recent past productivity of investigators at qualifying institutions, among other possible criteria. Disengaging overhead from individual grants would encourage administrators to value grants for the science those grants enable (as opposed to the overhead they bring), while allocating overhead based on institutions’ aggregate scientific productivity would motivate universities to help their investigators produce good science. Funding agencies often have pragmatic reasons to emphasize the meritocratic nature of their award processes. However, our model also suggests that downplaying elements of a funding competition' s structure that introduce randomness to funding decisions can increase scientific waste. When applicants fail to recognize the degree to which the contest is already a lottery, they will overinvest effort in preparing proposals, to the detriment of science. This model does not account for all of the costs or scientific benefits of a proposal competition, including the costs of administering the competition, the time lost to reviewing grant proposals, or the benefit of building scientific community through convening a review panel. Nonetheless, we suggest that the direct value of the science supported by funding awards and the disutility costs of preparing grant proposals are the predominant scientific benefits and costs of the usual proposal competition [13,30], and provide a useful starting point for a more detailed accounting. Our model also makes several simplifying assumptions, each of which may provide scope for interesting future work. First, researchers pay a time cost to prepare a proposal but receive money if the proposal is funded. In our model, we have converted both time and money into scientific productivity, in order to place both on a common footing. To be more explicit, though, scientific productivity requires both time and money (among other resources), and researchers may have vastly different needs for both. In S1 Text, we show that our model can be formally extended to encompass researchers' different needs for time and money if the marginal rate of technical substitution (that is, the rate at which time and money can be exchanged without altering scientific productivity) is exactly correlated with the project' s scientific value. Our main results still hold in this case, as long as researchers with the best ideas do not value time so greatly that they write the weakest proposals. A more general exploration of researchers' heterogeneous needs for time and money—and of how researchers may adjust their portfolio of scientific activities when time or money is scarce—provide ample opportunity for future work. Second, our model assumes that the distribution of the scientific value (v) across possible projects is exogeneous to the structure of the funding competition. This may not be the case if, for instance, a partial lottery encourages participation by investigators with unconventional views, reduces the psychological stigma of previous rejection [16], or discourages investigators either who have succeeded under the traditional proposal competition format or who perceive a lottery as riskier. In reality, such feedback loops may endogenize the distribution of v. Third, our model does not consider the savings that may accrue to investigators if they can submit a revised version of a rejected proposal to a different or subsequent competition. To a first approximation, submissions to multiple funders have the effect of increasing p, which can then be interpreted more generally as the proportion of ideas that get funded across all available funding programs. Iterations of revision and resubmission to the same funding program are likely to have more complex effects on efficiency and waste. Finally, our model is silent regarding whether many small or few large grants will promote scientific progress most efficiently, and is likewise silent about the factors that will influence this comparison. To be sure, much more can be done to embellish this model. However, the qualitative results—that proposal competitions become increasingly inefficient as paylines drop and that professional pressure on investigators to pursue funding exacerbates these inefficiencies—are inherent to the structure of contests. Partial lotteries and contests that reward past success present radical alternatives for allocating funds and are sure to be controversial. Nevertheless, whatever their other merits and drawbacks, these alternatives could restore efficiency in distributing funds that has been lost as those funds have become increasingly scarce.
The grant proposal system compels researchers to devote substantial time to writing proposals that could have instead been used to do science. Here, we use the economic theory of contests to show that as fewer grants are funded, the value of the science that researchers forgo while preparing proposals can approach or exceed the value of the science that the funding program supports. As a result, much of the scientific impact of the funding program is squandered. Unfortunately, increased waste and reduced efficiency is inevitable in a grant proposal competition when the number of awards is small. How can scarce funds be allocated efficiently, then? As one alternative, we show that a partial lottery that selects proposals for funding randomly from among those that pass a qualifying standard can restore lost efficiency by reducing investigators' incentives to invest heavily in preparing proposals. Lotteries could also improve efficiency by compelling administrators to de-emphasize grant success as a primary measure of professional achievement. If lotteries are politically untenable, another remedy would be to fund researchers based on their previous research successes, although in such a way that avoids establishing barriers to entry for junior scientists or scientists from historically underrepresented demographic groups.
research funding
bibliometrics
social sciences
neuroscience
research design
scientists
cognitive psychology
mathematics
probability distribution
research grants
science and technology workforce
research and analysis methods
meta-research article
behavior
economics
research assessment
probability theory
people and places
professions
psychology
survey research
science policy
careers in research
motivation
population groupings
biology and life sciences
physical sciences
cognitive science
Contest models highlight inherent inefficiencies of scientific funding competitions