<story><title>Plan to replicate 50 high-impact cancer papers shrinks to just 18</title><url>http://www.sciencemag.org/news/2018/07/plan-replicate-50-high-impact-cancer-papers-shrinks-just-18</url></story><parent_chain><item><author>sytelus</author><text>Every academic cries about this but no one wants to accept an &lt;i&gt;obvious&lt;/i&gt; solution: Add reprodicibility criteria for peer review. Right now even Tier 1 conferences do not have this criteria in their peer review system. The reviewer simply can&amp;#x27;t reject paper because authors didn&amp;#x27;t provided reproducibility information!&lt;p&gt;This is truely a plague in research right now. I&amp;#x27;ve came across quite a few instances where authors told me that their experiments weren&amp;#x27;t reproducible even for them! They do not note this in paper because ultimately everyone needs to show some result for the funding they received (i.e. &amp;quot;published paper&amp;quot;). They obviously never want to share any code, data files, hardware etc. On one instance, author wrote me back that they can&amp;#x27;t share code with me because they lost all the code because their hard drive crashed! Reproducibility is a fundamental tenant of doing scientific work and this is actively and completely ignored in current peer review system.&lt;p&gt;I think conference chairs needs to take stand on this. We get now 4X to 8X papers in tier 1 conferences. Reproducibility could be a great filter when area chairs are scrambling to find reasons to reject papers. Sure, there will be papers where very specialized hardware or internal infrastructure of 10,000 computers were used. But those papers would be great for Tier 2 conferences.</text></item></parent_chain><comment><author>setgree</author><text>Hi sytelus, &lt;i&gt;Nature&lt;/i&gt; is piloting (analytical&amp;#x2F;computational) reproducibility as part of peer review with, disclosure, the company I work for, Code Ocean:&lt;p&gt;&lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;blogs.nature.com&amp;#x2F;ofschemesandmemes&amp;#x2F;2018&amp;#x2F;08&amp;#x2F;01&amp;#x2F;nature-research-journals-trial-new-tools-to-enhance-code-peer-review-and-publication&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;blogs.nature.com&amp;#x2F;ofschemesandmemes&amp;#x2F;2018&amp;#x2F;08&amp;#x2F;01&amp;#x2F;nature-...&lt;/a&gt;&lt;p&gt;A few political science journals also hire staff to make sure that the code reproduce the results in the article:&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;ajps.org&amp;#x2F;ajps-replication-policy&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;ajps.org&amp;#x2F;ajps-replication-policy&amp;#x2F;&lt;/a&gt;&lt;p&gt;A nice writeup of the AJPS&amp;#x27;s experiences:&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.insidehighered.com&amp;#x2F;blogs&amp;#x2F;rethinking-research&amp;#x2F;should-journals-be-responsible-reproducibility&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.insidehighered.com&amp;#x2F;blogs&amp;#x2F;rethinking-research&amp;#x2F;sho...&lt;/a&gt;</text></comment>
<story><title>Plan to replicate 50 high-impact cancer papers shrinks to just 18</title><url>http://www.sciencemag.org/news/2018/07/plan-replicate-50-high-impact-cancer-papers-shrinks-just-18</url></story><parent_chain><item><author>sytelus</author><text>Every academic cries about this but no one wants to accept an &lt;i&gt;obvious&lt;/i&gt; solution: Add reprodicibility criteria for peer review. Right now even Tier 1 conferences do not have this criteria in their peer review system. The reviewer simply can&amp;#x27;t reject paper because authors didn&amp;#x27;t provided reproducibility information!&lt;p&gt;This is truely a plague in research right now. I&amp;#x27;ve came across quite a few instances where authors told me that their experiments weren&amp;#x27;t reproducible even for them! They do not note this in paper because ultimately everyone needs to show some result for the funding they received (i.e. &amp;quot;published paper&amp;quot;). They obviously never want to share any code, data files, hardware etc. On one instance, author wrote me back that they can&amp;#x27;t share code with me because they lost all the code because their hard drive crashed! Reproducibility is a fundamental tenant of doing scientific work and this is actively and completely ignored in current peer review system.&lt;p&gt;I think conference chairs needs to take stand on this. We get now 4X to 8X papers in tier 1 conferences. Reproducibility could be a great filter when area chairs are scrambling to find reasons to reject papers. Sure, there will be papers where very specialized hardware or internal infrastructure of 10,000 computers were used. But those papers would be great for Tier 2 conferences.</text></item></parent_chain><comment><author>dekhn</author><text>If a researcher told me they couldn&amp;#x27;t provide primary data because their hard drive crashed, I&amp;#x27;d contact their dean and the editor of the journal that published the article, and let them know.  Researchers have a duty (unfortunately not well codified) to retain primary source data, and &amp;quot;hard drive crash&amp;quot; isn&amp;#x27;t an excuse (it means they didn&amp;#x27;t take even basic backup and archival measures).</text></comment>