<story><title>New academic journal only publishes &apos;unsurprising&apos; research rejected by others</title><url>https://www.cbc.ca/radio/asithappens/as-it-happens-thursday-edition-1.5146761/new-academic-journal-only-publishes-unsurprising-research-rejected-by-others-1.5146765</url></story><parent_chain><item><author>mikk14</author><text>I disagree. &lt;i&gt;Negative&lt;/i&gt; results are important. &lt;i&gt;Null&lt;/i&gt; results are of very limited interest. The two are worlds apart.&lt;p&gt;A null result simply means you tried something and it didn&amp;#x27;t work. But you don&amp;#x27;t know why. You haven&amp;#x27;t proven it didn&amp;#x27;t work. There are literally millions of reasons why something might not work. For instance, you could try to use compound X to cure disease Y, observe no effect, and conclude that X doesn&amp;#x27;t cure Y. But what if somewhere in the process of making X you made an uncaught mistake and you instead used X&amp;#x27;?&lt;p&gt;A negative result means that you tried something and you came to the proven conclusion it doesn&amp;#x27;t work. This is, crucially, as hard to obtain as a positive result. In my example, it would imply a much longer process than simply &amp;quot;apply X, see no effect in Y, make a few robustness checks, done&amp;quot;.&lt;p&gt;You could say &amp;quot;Well, publish the null anyway, somebody will catch the mistake&amp;quot;. Unlikely. There are already so many papers out there that keeping up is impossible. If we were publishing also null results this number will grow tenfold at the very least. Nobody could possibly check everything. They will see a paper &amp;quot;X doesn&amp;#x27;t cure Y&amp;quot; and call it knowledge, stifling a possible cure virtually forever.&lt;p&gt;Am I splitting hairs? Perhaps. But I think HN prizes itself to be a scientifically minded community, and thus it has a mandate to use terms correctly. Confusing &amp;quot;null&amp;quot; with &amp;quot;negative&amp;quot; is a sin.&lt;p&gt;I hope one day I&amp;#x27;ll find a way to strongly and passionately argue against the &amp;quot;null results are as important as positive results&amp;quot; position. It is a bad meme. Charitably, I consider it most of the times a honest mistake. But sometimes it gives me the impression it is a cheap trick used by people to erode the reputation of academia.</text></item><item><author>owenshen24</author><text>That sucks to hear. Null results are important, as you say, if only to dissuade others from doing the same.&lt;p&gt;See also the &amp;quot;file-drawer problem&amp;quot; (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Publication_bias&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Publication_bias&lt;/a&gt;). Also, with regards to the incentives in the field and the lack of null results, there&amp;#x27;s always Ioannidis&amp;#x27;s classic work (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;journals.plos.org&amp;#x2F;plosmedicine&amp;#x2F;article?id=10.1371&amp;#x2F;journal.pmed.0020124&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;journals.plos.org&amp;#x2F;plosmedicine&amp;#x2F;article?id=10.1371&amp;#x2F;jo...&lt;/a&gt;).</text></item><item><author>kraetzin</author><text>Half of my PhD thesis is considered &amp;quot;unpublishable&amp;quot; because, after doing the work, my supervisors felt it&amp;#x27;s actually &amp;quot;unsurprising&amp;quot; that it didn&amp;#x27;t work out. We took methods that had been exploited to improve on previous results for over a decade to their logical extreme, and found that this method no longer leads to improvements. After doing the work it seems obvious. A paper on the subject would almost be considered uninteresting, and a high ranking journal would ignore it (which is why it&amp;#x27;s considered &amp;quot;unpublishable&amp;quot;). However, nobody has published this information, and it would help others to not make the same mistake.&lt;p&gt;I wonder how many times similar &amp;quot;mistakes&amp;quot; have been made by PhD students across disciplines.</text></item></parent_chain><comment><author>learnstats2</author><text>I understand that people generally mean &amp;#x27;null results&amp;#x27; here, by your definition, when they say &amp;#x27;null results&amp;#x27;. That&amp;#x27;s the intent.&lt;p&gt;Null results are &lt;i&gt;also important&lt;/i&gt;.&lt;p&gt;Suppression of null results allows for p-hacking and confirmation biases to creep into research, and greatly reduces the power of literature reviews.</text></comment>
<story><title>New academic journal only publishes &apos;unsurprising&apos; research rejected by others</title><url>https://www.cbc.ca/radio/asithappens/as-it-happens-thursday-edition-1.5146761/new-academic-journal-only-publishes-unsurprising-research-rejected-by-others-1.5146765</url></story><parent_chain><item><author>mikk14</author><text>I disagree. &lt;i&gt;Negative&lt;/i&gt; results are important. &lt;i&gt;Null&lt;/i&gt; results are of very limited interest. The two are worlds apart.&lt;p&gt;A null result simply means you tried something and it didn&amp;#x27;t work. But you don&amp;#x27;t know why. You haven&amp;#x27;t proven it didn&amp;#x27;t work. There are literally millions of reasons why something might not work. For instance, you could try to use compound X to cure disease Y, observe no effect, and conclude that X doesn&amp;#x27;t cure Y. But what if somewhere in the process of making X you made an uncaught mistake and you instead used X&amp;#x27;?&lt;p&gt;A negative result means that you tried something and you came to the proven conclusion it doesn&amp;#x27;t work. This is, crucially, as hard to obtain as a positive result. In my example, it would imply a much longer process than simply &amp;quot;apply X, see no effect in Y, make a few robustness checks, done&amp;quot;.&lt;p&gt;You could say &amp;quot;Well, publish the null anyway, somebody will catch the mistake&amp;quot;. Unlikely. There are already so many papers out there that keeping up is impossible. If we were publishing also null results this number will grow tenfold at the very least. Nobody could possibly check everything. They will see a paper &amp;quot;X doesn&amp;#x27;t cure Y&amp;quot; and call it knowledge, stifling a possible cure virtually forever.&lt;p&gt;Am I splitting hairs? Perhaps. But I think HN prizes itself to be a scientifically minded community, and thus it has a mandate to use terms correctly. Confusing &amp;quot;null&amp;quot; with &amp;quot;negative&amp;quot; is a sin.&lt;p&gt;I hope one day I&amp;#x27;ll find a way to strongly and passionately argue against the &amp;quot;null results are as important as positive results&amp;quot; position. It is a bad meme. Charitably, I consider it most of the times a honest mistake. But sometimes it gives me the impression it is a cheap trick used by people to erode the reputation of academia.</text></item><item><author>owenshen24</author><text>That sucks to hear. Null results are important, as you say, if only to dissuade others from doing the same.&lt;p&gt;See also the &amp;quot;file-drawer problem&amp;quot; (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Publication_bias&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Publication_bias&lt;/a&gt;). Also, with regards to the incentives in the field and the lack of null results, there&amp;#x27;s always Ioannidis&amp;#x27;s classic work (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;journals.plos.org&amp;#x2F;plosmedicine&amp;#x2F;article?id=10.1371&amp;#x2F;journal.pmed.0020124&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;journals.plos.org&amp;#x2F;plosmedicine&amp;#x2F;article?id=10.1371&amp;#x2F;jo...&lt;/a&gt;).</text></item><item><author>kraetzin</author><text>Half of my PhD thesis is considered &amp;quot;unpublishable&amp;quot; because, after doing the work, my supervisors felt it&amp;#x27;s actually &amp;quot;unsurprising&amp;quot; that it didn&amp;#x27;t work out. We took methods that had been exploited to improve on previous results for over a decade to their logical extreme, and found that this method no longer leads to improvements. After doing the work it seems obvious. A paper on the subject would almost be considered uninteresting, and a high ranking journal would ignore it (which is why it&amp;#x27;s considered &amp;quot;unpublishable&amp;quot;). However, nobody has published this information, and it would help others to not make the same mistake.&lt;p&gt;I wonder how many times similar &amp;quot;mistakes&amp;quot; have been made by PhD students across disciplines.</text></item></parent_chain><comment><author>tralarpa</author><text>&amp;gt; Negative results are important. Null results are of very limited interest.&lt;p&gt;Correct. There is a highly cited paper in CS where the author showed that a mathematical model that was widely used in research didn&amp;#x27;t actually work (anymore) in reality. That paper was the starting point of a lot of new research in that field.</text></comment>