<story><title>It’s not just p=0.048 vs. p=0.052</title><url>https://statmodeling.stat.columbia.edu/2019/09/06/__trashed-2/</url></story><parent_chain></parent_chain><comment><author>Symmetry</author><text>Put a Number on It! did a piece a while ago going through some psychology pieces that were part of a replication effort.  They found that half failed to replicate but that people in a betting market could often tell which ones were going to replicate or not.  The author also did a blind test himself and was also able to guess which ones would replicate.  He laid out several rules of thumb, most significantly to the article &lt;i&gt;Jacob’s Rule of Anti-Significance: A result with a p-value just above 0.05 could well be true. A result with a p-value just below 0.05 is almost certainly false.&lt;/i&gt;&lt;p&gt;&lt;i&gt;More importantly, p=0.06 means that the researchers are honest. They could have easily p-hacked the results below 0.05 but chose not to. The opposite is true when p=0.049.&lt;/i&gt;&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;putanumonit.com&amp;#x2F;2018&amp;#x2F;09&amp;#x2F;07&amp;#x2F;the-scent-of-bad-psychology&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;putanumonit.com&amp;#x2F;2018&amp;#x2F;09&amp;#x2F;07&amp;#x2F;the-scent-of-bad-psycholo...&lt;/a&gt;</text></comment>
<story><title>It’s not just p=0.048 vs. p=0.052</title><url>https://statmodeling.stat.columbia.edu/2019/09/06/__trashed-2/</url></story><parent_chain></parent_chain><comment><author>twelfthnight</author><text>I think the problem with p-values is that it trains us to think about uncertainty without nuance. It hides the inherent trade off between the cost of taking on risk and the cost of reducing uncertainty, since it sets the threshold at p=0.05.  Taken to the extreme, with a large enough sample we can nearly always find significant differences between populations, the difference will just be very small and n size will be enormous.&lt;p&gt;Recently I worked with a client to interpret results from an A&amp;#x2F;B test where A performed better than B with 85% confidence (based on credible intervals, accounting for multiple comparisons). We therefore recommended A.  In a group phone call, the client told her colleagues that our company doesn&amp;#x27;t know what we&amp;#x27;re talking about because 85% confidence of a difference isn&amp;#x27;t statistically significant (i.e. isn&amp;#x27;t 95% confident).  We lost their business.&lt;p&gt;This was a shame because gathering the data for the experiment was expensive and the downside of making the wrong choice was low.  It is often the case that taking on more risk makes more sense than hitting diminishing returns on shrinking p-values with extra sample.</text></comment>