<story><title>How to Read a Paper (2016) [pdf]</title><url>https://blizzard.cs.uwaterloo.ca/keshav/home/Papers/data/07/paper-reading.pdf</url></story><parent_chain><item><author>TrackerFF</author><text>I know there are a lot of ML researchers and practitioners here - and I have unfortunately only a very shallow experience with reading ML papers, more so with the recent output.&lt;p&gt;But, I have to ask, how do you get a feel that the content actually looks correct, and not just only quack? The improvements are usually in the 1% range, from old models to new models, and the models are complex. More often than not also lacking code, implementation &amp;#x2F; experiment procedures, etc.&lt;p&gt;Basically, I have no idea if the paper is reproducible, if the results are cherry picked from hundreds &amp;#x2F; thousands of runs, if the paper is just cleverly disguised BS with pumped up numbers to get grants, and so on.&lt;p&gt;As it is right now, I can only rely on expert assurance from those that peer review these papers - but even then, in the back of my mind, I&amp;#x27;m wondering if they&amp;#x27;ve had time to rigorously review a paper. The output of ML &amp;#x2F; AI papers these days is staggering, and the systems are so complex that I&amp;#x27;d be impressed if some single post. doc or researcher would have time to reproduce results.</text></item></parent_chain><comment><author>dennybritz</author><text>You will never know the &amp;quot;generative process&amp;quot; of a paper, which makes it nearly impossible to properly evaluate it, even as an expert reviewer. If you are not a reviewer (or working on a competing paper), you are much better off NOT reading the newest papers. Instead, rely on the best proxy metric of all: Time. Wait until the paper has been battle-tested. See if people on social media are talking about it, what critics are saying, wait for reproduction results and open-source implementations, conference acceptances, and citations and comparisons. These give you a much better idea of the validity of the paper than its content can.&lt;p&gt;When reading the newest ML papers, I found it useful to not judge them, but instead use them as inspiration. Forget about the results. The paper may contain interesting ideas or viewpoints you didn&amp;#x27;t consider before, and those are probably much more valuable than the result table.</text></comment>
<story><title>How to Read a Paper (2016) [pdf]</title><url>https://blizzard.cs.uwaterloo.ca/keshav/home/Papers/data/07/paper-reading.pdf</url></story><parent_chain><item><author>TrackerFF</author><text>I know there are a lot of ML researchers and practitioners here - and I have unfortunately only a very shallow experience with reading ML papers, more so with the recent output.&lt;p&gt;But, I have to ask, how do you get a feel that the content actually looks correct, and not just only quack? The improvements are usually in the 1% range, from old models to new models, and the models are complex. More often than not also lacking code, implementation &amp;#x2F; experiment procedures, etc.&lt;p&gt;Basically, I have no idea if the paper is reproducible, if the results are cherry picked from hundreds &amp;#x2F; thousands of runs, if the paper is just cleverly disguised BS with pumped up numbers to get grants, and so on.&lt;p&gt;As it is right now, I can only rely on expert assurance from those that peer review these papers - but even then, in the back of my mind, I&amp;#x27;m wondering if they&amp;#x27;ve had time to rigorously review a paper. The output of ML &amp;#x2F; AI papers these days is staggering, and the systems are so complex that I&amp;#x27;d be impressed if some single post. doc or researcher would have time to reproduce results.</text></item></parent_chain><comment><author>ssivark</author><text>&amp;gt; The improvements are usually in the 1% range, from old models to new models, and the models are complex. More often than not also lacking code, implementation &amp;#x2F; experiment procedures, etc. [...] no idea if the paper is reproducible, if the results are cherry picked from hundreds &amp;#x2F; thousands of runs, if the paper is just cleverly disguised BS with pumped up numbers to get grants, and so on.&lt;p&gt;Personally, I&amp;#x27;m bearish about most deep learning papers for this reason.&lt;p&gt;I&amp;#x27;m not driven by a particular task&amp;#x2F;problem, so when I&amp;#x27;m reading ML papers, it is primarily for new insights and ideas. Correspondingly, I prefer to read papers which have new perspectives on the problem (irrespective of whether they achieve SOTA performance). From what I&amp;#x27;ve seen, most of the interesting (to me) ideas come from slightly adjacent fields. I care far more about interesting &amp;amp; elegant ideas, and benchmarks to just sanity-check that the nice idea can also be made to work in practice.&lt;p&gt;As for the obsession with benchmark numbers, I can only quote Mark Twain: &lt;i&gt;“Most people use statistics like a drunk man uses a lamppost; more for support than illumination.”&lt;/i&gt;</text></comment>