<story><title>It will be much harder to call findings ‘significant’ if a team gets its way</title><url>http://www.sciencemag.org/news/2017/07/it-will-be-much-harder-call-new-findings-significant-if-team-gets-its-way</url></story><parent_chain><item><author>elsherbini</author><text>There are two legitimate ways to get lower p-values: You can have a larger effect size, or you can have a larger number of samples[0]. Of course, you can&amp;#x27;t change the effect size, so this would lead to larger necessary samples sizes to study smaller effects.&lt;p&gt;I think in general though, at least in biology, people are waking up to the fact that p-values aren&amp;#x27;t magical, and that having a really small p-value isn&amp;#x27;t a goal in and of itself. It is, however, necessary to do some statistics on your data to get it published, but the p-value is just checking a box more than being used as a tool for discovery.&lt;p&gt;Yesterday a cool dataset was released from Jeff Leek which has over 3.6 million p-values from scientific literature[1]. The distribution is fun to look at by discipline[2].&lt;p&gt;[0] &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;rpsychologist.com&amp;#x2F;d3&amp;#x2F;NHST&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;rpsychologist.com&amp;#x2F;d3&amp;#x2F;NHST&amp;#x2F;&lt;/a&gt;&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;jtleek&amp;#x2F;tidypvals&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;jtleek&amp;#x2F;tidypvals&lt;/a&gt;&lt;p&gt;[2] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;twitter.com&amp;#x2F;drob&amp;#x2F;status&amp;#x2F;890260541876338690&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;twitter.com&amp;#x2F;drob&amp;#x2F;status&amp;#x2F;890260541876338690&lt;/a&gt;</text></item></parent_chain><comment><author>Amygaz</author><text>I just want to buffer a little bit what you said about &amp;quot;the p-value is just checking a box more than being used as a tool for discovery.&amp;quot;, but we are likely on the same page.&lt;p&gt;Sometimes, it is necessary to use a significance test in order to convince the reader that your conclusion based on your interpretation of your observations is sensical, which would be &amp;quot;checking a box&amp;quot;.&lt;p&gt;Other times, and as a frequent reviewer I do see that, infrequently, but still, where the authors see something where the only thing to see seems completely anecdotal (small N, large or fake error bars, no significance or ranking test), or to support their innacurate interpretation of a phenomenon (I see that with young and inexperienced graduate student in a young and inexperienced PI&amp;#x27;s lab). In these cases, having a proper experimental setup, sample size, and analysis, is part of the discovery.&lt;p&gt;At the end of the day, it is important to make sure that science stays evidence-based and statistic is one of the tools that can assist, but only assist, in doing that.</text></comment>
<story><title>It will be much harder to call findings ‘significant’ if a team gets its way</title><url>http://www.sciencemag.org/news/2017/07/it-will-be-much-harder-call-new-findings-significant-if-team-gets-its-way</url></story><parent_chain><item><author>elsherbini</author><text>There are two legitimate ways to get lower p-values: You can have a larger effect size, or you can have a larger number of samples[0]. Of course, you can&amp;#x27;t change the effect size, so this would lead to larger necessary samples sizes to study smaller effects.&lt;p&gt;I think in general though, at least in biology, people are waking up to the fact that p-values aren&amp;#x27;t magical, and that having a really small p-value isn&amp;#x27;t a goal in and of itself. It is, however, necessary to do some statistics on your data to get it published, but the p-value is just checking a box more than being used as a tool for discovery.&lt;p&gt;Yesterday a cool dataset was released from Jeff Leek which has over 3.6 million p-values from scientific literature[1]. The distribution is fun to look at by discipline[2].&lt;p&gt;[0] &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;rpsychologist.com&amp;#x2F;d3&amp;#x2F;NHST&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;rpsychologist.com&amp;#x2F;d3&amp;#x2F;NHST&amp;#x2F;&lt;/a&gt;&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;jtleek&amp;#x2F;tidypvals&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;jtleek&amp;#x2F;tidypvals&lt;/a&gt;&lt;p&gt;[2] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;twitter.com&amp;#x2F;drob&amp;#x2F;status&amp;#x2F;890260541876338690&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;twitter.com&amp;#x2F;drob&amp;#x2F;status&amp;#x2F;890260541876338690&lt;/a&gt;</text></item></parent_chain><comment><author>BeetleB</author><text>&amp;gt;There are two legitimate ways to get lower p-values: You can have a larger effect size, or you can have a larger number of samples[0]. Of course, you can&amp;#x27;t change the effect size, so this would lead to larger necessary samples sizes to study smaller effects.&lt;p&gt;Be careful with this. Larger sample sizes are more likely to give you a significant result even if one doesn&amp;#x27;t exist.&lt;p&gt;Say my null hypothesis is that X=100. The alternative hypothesis is X&amp;gt;100.&lt;p&gt;What if in reality X is really 100.5? Depending on the problem domain, this may well be the same as the null hypothesis. But a larger sample size is much more likely to give a significant result.&lt;p&gt;There are ways to fix this, but one should just be aware, though.</text></comment>