<story><title>Machine Learning Reproducibility Checklist [pdf]</title><url>https://www.cs.mcgill.ca/~jpineau/ReproducibilityChecklist.pdf</url></story><parent_chain><item><author>sillysaurusx</author><text>This checklist has some flaws. Most interesting results in ML have no proof.&lt;p&gt;For example, can you give a proof of superconvergence? What’s the exact learning rate that causes it, and why? Did you know that you can often get away with a high learning rate for a time, and then divergence happens? What’s the proof of that?&lt;p&gt;Give a proof that under all circumstances and wind conditions, lowering your airplane’s flaps by 5 degrees will help you land safely.&lt;p&gt;Also, what about datasets that you’re not allowed to release? I personally despise such datasets, but I found myself in the ironic position of having a 10GB dataset dropped in my lap that was a perfect fit for my current project. Unfortunately it wasn’t until after training was mostly complete that we realized we hadn’t asked whether the author was comfortable releasing it, and indeed the answer was no. So what to do? Just don’t talk about it?&lt;p&gt;I guess the list is good as a set of ideals to aim for. I just wish some consideration was given that you often can’t meet all of those goals.&lt;p&gt;Most of OpenAI&amp;#x27;s work would be excluded by this checklist. I don&amp;#x27;t think anyone would argue that OpenAI doesn&amp;#x27;t do important work, and that their results are in some sense reproducible.</text></item></parent_chain><comment><author>TeMPOraL</author><text>&amp;gt; &lt;i&gt;Give a proof that under all circumstances and wind conditions, lowering your airplane’s flaps by 5 degrees will help you land safely.&lt;/i&gt;&lt;p&gt;My passing familiarity with aerodynamics and control theory suggests that you could derive a multidimentional shell in parameter space (of wind, temperature, airspeed and other conditions), to form an envelope within which the plane will behave predictably, so that you can prove whether lowering flaps by 5 degrees at a particular point will help you land safely. Accounting for model uncertainty, that envelope would likely be tighter than it could be if we knew our physics better, but that&amp;#x27;s still far better than a black box ML model that doesn&amp;#x27;t give you guarantees that similar inputs will lead to similar outputs (there&amp;#x27;s a mathematical formalism to this whose name escapes me now).</text></comment>
<story><title>Machine Learning Reproducibility Checklist [pdf]</title><url>https://www.cs.mcgill.ca/~jpineau/ReproducibilityChecklist.pdf</url></story><parent_chain><item><author>sillysaurusx</author><text>This checklist has some flaws. Most interesting results in ML have no proof.&lt;p&gt;For example, can you give a proof of superconvergence? What’s the exact learning rate that causes it, and why? Did you know that you can often get away with a high learning rate for a time, and then divergence happens? What’s the proof of that?&lt;p&gt;Give a proof that under all circumstances and wind conditions, lowering your airplane’s flaps by 5 degrees will help you land safely.&lt;p&gt;Also, what about datasets that you’re not allowed to release? I personally despise such datasets, but I found myself in the ironic position of having a 10GB dataset dropped in my lap that was a perfect fit for my current project. Unfortunately it wasn’t until after training was mostly complete that we realized we hadn’t asked whether the author was comfortable releasing it, and indeed the answer was no. So what to do? Just don’t talk about it?&lt;p&gt;I guess the list is good as a set of ideals to aim for. I just wish some consideration was given that you often can’t meet all of those goals.&lt;p&gt;Most of OpenAI&amp;#x27;s work would be excluded by this checklist. I don&amp;#x27;t think anyone would argue that OpenAI doesn&amp;#x27;t do important work, and that their results are in some sense reproducible.</text></item></parent_chain><comment><author>YeGoblynQueenne</author><text>The checklist item that asks for a proof is for theoretical claims. Empirical claims are supported by experiment. Theoretical claims must also be supported theoretically.&lt;p&gt;To put it plainly, if your paper has a &amp;quot;Framework&amp;quot; section that has some text titled &amp;quot;lemma&amp;quot; and &amp;quot;theorem&amp;quot; then you also need some text titled &amp;quot;proof&amp;quot;</text></comment>