<story><title>Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See</title><url>https://www.motherjones.com/kevin-drum/2018/11/chart-of-the-decade-why-you-shouldnt-trust-every-scientific-study-you-see/</url></story><parent_chain><item><author>epistasis</author><text>I 100% agree with the headline; &lt;i&gt;especially&lt;/i&gt; for research papers even more than clinical trials.&lt;p&gt;However, I&amp;#x27;m a bit puzzled by the weird direction the journalist ran with this, which is straight to his preconceived notions that are not that supported by the data he&amp;#x27;s looking at.&lt;p&gt;But there&amp;#x27;s a bit more to this than just that one chart. In addition to self-correction (e.g. beginning to require pre-registration of trials), science is somewhat additive. Is it not possible that the low-hanging fruit had been found earlier, in the 1970s-1990s, and the problem got harder? With the advances in treatments for cardiovascular disease couldn&amp;#x27;t the problem just be harder?&lt;p&gt;And look at that &amp;quot;harm&amp;quot; study. Turns out it&amp;#x27;s one of many from the Women&amp;#x27;s Health Initiative [1], and it&amp;#x27;s the test of whether hormone therapy for post-menopausal women causes heart problems: that is, it&amp;#x27;s not a test of a drug to prevent heart conditions, but a test of side effects.&lt;p&gt;How many other of these studies are like that: studies about effects on the heart, not trials for new drugs to treat the heart? How did the ratio change before and after 2000?&lt;p&gt;In any case, don&amp;#x27;t trust every popular news article you read about science, particularly if it&amp;#x27;s written by Kevin Drum and posted on Mother Jones.&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.whi.org&amp;#x2F;SitePages&amp;#x2F;WHI%20Home.aspx&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.whi.org&amp;#x2F;SitePages&amp;#x2F;WHI%20Home.aspx&lt;/a&gt;</text></item></parent_chain><comment><author>saalweachter</author><text>I&amp;#x27;ll go one stronger and say &amp;quot;You Shouldn&amp;#x27;t Trust Any One Scientific Study You See&amp;quot;.&lt;p&gt;Individual studies can be really interesting.  They&amp;#x27;re important for researchers to know about to inform their future work.  But any one study - even ones that are done honestly, with good methodology and sound foundations - can be just totally wrong.  There could be confounding factors you couldn&amp;#x27;t have known about that completely invalidate the result.  Your test subjects could be unusual in some way, your animal models could be a poor analogue for humans in this particular case, you could have just had really aberrant statistical flukes in your statistic sampling.&lt;p&gt;It&amp;#x27;s the &lt;i&gt;body&lt;/i&gt; of scientific research, the dozens, hundreds, thousands of studies stacked on top of each other that bring certainty.</text></comment>
<story><title>Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See</title><url>https://www.motherjones.com/kevin-drum/2018/11/chart-of-the-decade-why-you-shouldnt-trust-every-scientific-study-you-see/</url></story><parent_chain><item><author>epistasis</author><text>I 100% agree with the headline; &lt;i&gt;especially&lt;/i&gt; for research papers even more than clinical trials.&lt;p&gt;However, I&amp;#x27;m a bit puzzled by the weird direction the journalist ran with this, which is straight to his preconceived notions that are not that supported by the data he&amp;#x27;s looking at.&lt;p&gt;But there&amp;#x27;s a bit more to this than just that one chart. In addition to self-correction (e.g. beginning to require pre-registration of trials), science is somewhat additive. Is it not possible that the low-hanging fruit had been found earlier, in the 1970s-1990s, and the problem got harder? With the advances in treatments for cardiovascular disease couldn&amp;#x27;t the problem just be harder?&lt;p&gt;And look at that &amp;quot;harm&amp;quot; study. Turns out it&amp;#x27;s one of many from the Women&amp;#x27;s Health Initiative [1], and it&amp;#x27;s the test of whether hormone therapy for post-menopausal women causes heart problems: that is, it&amp;#x27;s not a test of a drug to prevent heart conditions, but a test of side effects.&lt;p&gt;How many other of these studies are like that: studies about effects on the heart, not trials for new drugs to treat the heart? How did the ratio change before and after 2000?&lt;p&gt;In any case, don&amp;#x27;t trust every popular news article you read about science, particularly if it&amp;#x27;s written by Kevin Drum and posted on Mother Jones.&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.whi.org&amp;#x2F;SitePages&amp;#x2F;WHI%20Home.aspx&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.whi.org&amp;#x2F;SitePages&amp;#x2F;WHI%20Home.aspx&lt;/a&gt;</text></item></parent_chain><comment><author>stilley2</author><text>I would say that not all research needs to be held to the pre-registering standard of clinical trials. Some research is more exploratory, and should be interpreted that way.</text></comment>