<story><title>A Worrying Analysis of Recent Neural Recommendation Approaches</title><url>https://arxiv.org/abs/1907.06902</url></story><parent_chain><item><author>mayank</author><text>Research code is generally an abomination, scraped together by MS&amp;#x2F;PhD students who may never have had any industry exposure to best practices, norms, testing, etc. There may also be the fear of releasing buggy code, which if&amp;#x2F;when discovered, will discredit the paper. Far safer to just not release the code, since it isn&amp;#x27;t mandated (which should absolutely 100% change)&lt;p&gt;Source: am former producer&amp;#x2F;publisher of abomination research code.</text></item><item><author>goostavos</author><text>Is there some background reason that academics generally don&amp;#x27;t provide code in their papers? Hell, in a lot of papers I&amp;#x27;ve tried looking at recently[0], there will be paragraphs and paragraphs of english text &lt;i&gt;describing&lt;/i&gt; an algorithm rather than even just plunking down some pseudo code.&lt;p&gt;It makes pulling out the actual content of the papers such a slog.&lt;p&gt;[0] exhibit A: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;ieeexplore.ieee.org&amp;#x2F;document&amp;#x2F;1017616&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;ieeexplore.ieee.org&amp;#x2F;document&amp;#x2F;1017616&lt;/a&gt;</text></item><item><author>nkurz</author><text>Summarizing, they tried replicating the results of 18 recent papers, and determined that because of lack of available code or data, they were only able to attempt replications of 7 of them.  Of these, they found that the &amp;quot;improvements&amp;quot; in 6 of them could also be achieved with a simpler non-machine learning approach (such as nearest neighbor). For the 7th (Mult-VAE), they were able to almost match the improvement  with better tuning of one of the baselines used in that paper.&lt;p&gt;Does this mean that the published results are garbage?  No!  But what does it mean?&lt;p&gt;First, don&amp;#x27;t miss the main claim:  less than half the published papers have sufficient code and data available for replication.   While one might argue exactly what the standards for availability should be, if journals and reviewers were to demand it, this number can definitely be improved.   Greater availability of data and code makes for better science.&lt;p&gt;Second, my guess is that it means that paper authors are using excess &amp;quot;puffery&amp;quot; in an attempt to improve the chances that their papers are accepted for publication.  They&amp;#x27;ve decided that using a weak baseline and claiming a 50% improvement is more likely to result in publication than using a realistic baseline and claiming a 0-5% improvement.   Unfortunately, they are probably right.&lt;p&gt;The useful change (in my opinion) would be to change the standards by which papers are judged to be publishable.  Reviewers should push back against weak baselines, but be more accepting of results that make modest claims.  You can still have a useful paper even if the degree of improvement is small --- or even nonexistent.  Maybe it will inspire someone else, maybe it&amp;#x27;s useful in an ensemble, or maybe it applies to a case where the baseline wouldn&amp;#x27;t.  It&amp;#x27;s the puffery that&amp;#x27;s the problem, not the publication.  Creating a system that rewards authors who more honestly evaluate their work would be an overall win.</text></item></parent_chain><comment><author>khawkins</author><text>I disagree that it is a lack of industry exposure.  &amp;quot;Best practices, norms, testing, etc.&amp;quot; is a waste of time when you develop well written code that doesn&amp;#x27;t produce good results.  After months of tweaking even a good foundation of code, it begins looking atrocious.  But you get graduated for publications based on good results, not for code that someone else can use.</text></comment>
<story><title>A Worrying Analysis of Recent Neural Recommendation Approaches</title><url>https://arxiv.org/abs/1907.06902</url></story><parent_chain><item><author>mayank</author><text>Research code is generally an abomination, scraped together by MS&amp;#x2F;PhD students who may never have had any industry exposure to best practices, norms, testing, etc. There may also be the fear of releasing buggy code, which if&amp;#x2F;when discovered, will discredit the paper. Far safer to just not release the code, since it isn&amp;#x27;t mandated (which should absolutely 100% change)&lt;p&gt;Source: am former producer&amp;#x2F;publisher of abomination research code.</text></item><item><author>goostavos</author><text>Is there some background reason that academics generally don&amp;#x27;t provide code in their papers? Hell, in a lot of papers I&amp;#x27;ve tried looking at recently[0], there will be paragraphs and paragraphs of english text &lt;i&gt;describing&lt;/i&gt; an algorithm rather than even just plunking down some pseudo code.&lt;p&gt;It makes pulling out the actual content of the papers such a slog.&lt;p&gt;[0] exhibit A: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;ieeexplore.ieee.org&amp;#x2F;document&amp;#x2F;1017616&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;ieeexplore.ieee.org&amp;#x2F;document&amp;#x2F;1017616&lt;/a&gt;</text></item><item><author>nkurz</author><text>Summarizing, they tried replicating the results of 18 recent papers, and determined that because of lack of available code or data, they were only able to attempt replications of 7 of them.  Of these, they found that the &amp;quot;improvements&amp;quot; in 6 of them could also be achieved with a simpler non-machine learning approach (such as nearest neighbor). For the 7th (Mult-VAE), they were able to almost match the improvement  with better tuning of one of the baselines used in that paper.&lt;p&gt;Does this mean that the published results are garbage?  No!  But what does it mean?&lt;p&gt;First, don&amp;#x27;t miss the main claim:  less than half the published papers have sufficient code and data available for replication.   While one might argue exactly what the standards for availability should be, if journals and reviewers were to demand it, this number can definitely be improved.   Greater availability of data and code makes for better science.&lt;p&gt;Second, my guess is that it means that paper authors are using excess &amp;quot;puffery&amp;quot; in an attempt to improve the chances that their papers are accepted for publication.  They&amp;#x27;ve decided that using a weak baseline and claiming a 50% improvement is more likely to result in publication than using a realistic baseline and claiming a 0-5% improvement.   Unfortunately, they are probably right.&lt;p&gt;The useful change (in my opinion) would be to change the standards by which papers are judged to be publishable.  Reviewers should push back against weak baselines, but be more accepting of results that make modest claims.  You can still have a useful paper even if the degree of improvement is small --- or even nonexistent.  Maybe it will inspire someone else, maybe it&amp;#x27;s useful in an ensemble, or maybe it applies to a case where the baseline wouldn&amp;#x27;t.  It&amp;#x27;s the puffery that&amp;#x27;s the problem, not the publication.  Creating a system that rewards authors who more honestly evaluate their work would be an overall win.</text></item></parent_chain><comment><author>dv_dt</author><text>I wonder if a journal could be established that emphasizes code integrated papers, something that might accept submissions as jupyter notebooks for example.&lt;p&gt;Edit: Perhaps one way to encourage this is via topic &amp;quot;reuse&amp;quot; papers, which present further detail on a previously published papers topic, but this time with code and more detailed discussion. It would at least help the situations in that the author gets some reuse of effort, but in a way that still shows new information and advances the field.</text></comment>