<story><title>We don&apos;t know how to fix science (2021)</title><url>https://www.worksinprogress.co/issue/we-dont-know-how-to-fix-science/</url></story><parent_chain></parent_chain><comment><author>mikkergp</author><text>Science does indeed have a lot of problems, but is the problem really that science is broken, or that our expectations of it are.&lt;p&gt;&amp;gt; Ideally, we’d like to measure the benefit provided by a study to society. We might ask: had this piece of research not been funded, would a given invention have been delayed? If so, for how long? And what was the impact of that on societal welfare? We could also try to estimate intermediate metrics of research usefulness, like whether a given basic research paper will end up being used as input to an application. It is an argument for humility, not epistemic nihilism.&lt;p&gt;The whole promise of basic science research is that it may be valuable in ways that are immeasurable, or certainly uncapitalizable.  One major issue with science is that we&amp;#x27;re always hoping that it will say more than it should.  I would take a more mindfulness-y approach to science.  You do science because it is a good end, and if you do it for that reason, good things will come.  If you chase &amp;quot;good science&amp;quot; you get the problems we have.</text></comment>
<story><title>We don&apos;t know how to fix science (2021)</title><url>https://www.worksinprogress.co/issue/we-dont-know-how-to-fix-science/</url></story><parent_chain></parent_chain><comment><author>fiprofessor</author><text>I see a lot of appeals to &amp;quot;fund people, not projects&amp;quot;, but the main issue I have with it is that I think it strongly biases people to select &amp;quot;super stars&amp;quot; who have impressive credentials &amp;#x2F; come from high status organizations. Of course, there&amp;#x27;s already a lot of &amp;quot;rich get richer&amp;quot; effects in science, as the article points out, but I think it would be nice to try to shift &lt;i&gt;away&lt;/i&gt; from those things, not increase them. (Besides, isn&amp;#x27;t the resounding message in academia these days supposed to be that assessing people is full of biases that we ought to avoid? Of course you always assess the people behind a grant application to some extent, but focusing on the project would seem to be better in terms of avoiding these biases.)&lt;p&gt;The other alternative of &amp;quot;funding lotteries&amp;quot; is more suspect. As the article summarizes, the argument for lotteries is:&lt;p&gt;&amp;gt; &lt;i&gt;Advocates of lotteries make two key critiques: a) the current system forces researchers to spend a lot of time preparing grants; and b) peer reviewers cannot reliably identify “good” grant applications. They claim that a lottery system would reduce the time spent on review (because reviewers would mostly skim the proposals to check for minimal scientific robustness) as well as the time spent on preparing proposals (because there would be less of an incentive to meticulously craft proposals, given that no matter how detailed and well written they are, they are going to be chosen at random).&lt;/i&gt;&lt;p&gt;Of course (a) is true, but as the article suggests, the evidence for (b) is a bit shaky. Moreover, we should have strong priors against (b). To be sure, reviewing is not perfect, far from it! But, scientists must at some level be able to judge the future prospects of work, or else they wouldn&amp;#x27;t be able to make any fruitful decisions about what research to conduct. So the primary plausible way I see that (b) could perhaps be true for a given pool of applications is if all of the ones in the pool pass some thresh-hold bar for quality that makes it hard to further distinguish between them. It&amp;#x27;s not clear to me that that quality level would be maintained if we move to a lottery.</text></comment>