<story><title>The main trick in machine learning</title><url>http://edinburghhacklab.com/2013/12/the-main-trick-in-machine-learning/</url></story><parent_chain><item><author>joe_the_user</author><text>&lt;i&gt;In order to make it tractable, you pick a finite model space, train it on finite data, and use a finite algorithm to find the best choice inside of that space. That means you can fail in three ways---you can over-constrain your model space so that the true model cannot be found, you can underpower your search so that you have less an ability to discern the best model in your chosen model space, and you can terminate your search early and fail to reach that point entirely.&lt;/i&gt;&lt;p&gt;It seems like you can &amp;quot;mis-power&amp;quot; your model also.&lt;p&gt;For example, the Ptolemaic system could approximate the movement of the planets to any degree if you added enough &amp;quot;wheels within wheels&amp;quot; but since these were &amp;quot;the wrong wheels&amp;quot;, the necessary wheels grew without bounds to achieve reasonable approximation over time.</text></item><item><author>tel</author><text>A professor of mine stated it very well. If you can imagine that there is a &lt;i&gt;true&lt;/i&gt; model somewhere out in infinitely large model space then ML is just the search for that model.&lt;p&gt;In order to make it tractable, you pick a finite model space, train it on finite data, and use a finite algorithm to find the best choice inside of that space. That means you can fail in three ways---you can over-constrain your model space so that the true model cannot be found, you can underpower your search so that you have less an ability to discern the best model in your chosen model space, and you can terminate your search early and fail to reach that point entirely.&lt;p&gt;Almost all error in ML can be seen nicely in this model. In particular here, those who do not remember to optimize validation accuracy are often making their model space so large (overfitting) at the cost of having too little data to power the search within it.&lt;p&gt;Devroye, Gyorfi, and Lugosi (&lt;a href=&quot;http://www.amazon.com/Probabilistic-Recognition-Stochastic-Modelling-Probability/dp/0387946187&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;www.amazon.com&amp;#x2F;Probabilistic-Recognition-Stochastic-M...&lt;/a&gt;) have a really great picture of this in their book.</text></item></parent_chain><comment><author>tel</author><text>Also, to add, when DGL bring this kind of mental model up they do so to motivate a kind of semi-parametric modeling where the design space changes progressively to move closer to the true model without growing so quickly as to make inference unstable. The problem being, of course, that this causes your algorithm run time to blow out to something cubic, I think, and so you have a beautiful model that loses out on search error.</text></comment>
<story><title>The main trick in machine learning</title><url>http://edinburghhacklab.com/2013/12/the-main-trick-in-machine-learning/</url></story><parent_chain><item><author>joe_the_user</author><text>&lt;i&gt;In order to make it tractable, you pick a finite model space, train it on finite data, and use a finite algorithm to find the best choice inside of that space. That means you can fail in three ways---you can over-constrain your model space so that the true model cannot be found, you can underpower your search so that you have less an ability to discern the best model in your chosen model space, and you can terminate your search early and fail to reach that point entirely.&lt;/i&gt;&lt;p&gt;It seems like you can &amp;quot;mis-power&amp;quot; your model also.&lt;p&gt;For example, the Ptolemaic system could approximate the movement of the planets to any degree if you added enough &amp;quot;wheels within wheels&amp;quot; but since these were &amp;quot;the wrong wheels&amp;quot;, the necessary wheels grew without bounds to achieve reasonable approximation over time.</text></item><item><author>tel</author><text>A professor of mine stated it very well. If you can imagine that there is a &lt;i&gt;true&lt;/i&gt; model somewhere out in infinitely large model space then ML is just the search for that model.&lt;p&gt;In order to make it tractable, you pick a finite model space, train it on finite data, and use a finite algorithm to find the best choice inside of that space. That means you can fail in three ways---you can over-constrain your model space so that the true model cannot be found, you can underpower your search so that you have less an ability to discern the best model in your chosen model space, and you can terminate your search early and fail to reach that point entirely.&lt;p&gt;Almost all error in ML can be seen nicely in this model. In particular here, those who do not remember to optimize validation accuracy are often making their model space so large (overfitting) at the cost of having too little data to power the search within it.&lt;p&gt;Devroye, Gyorfi, and Lugosi (&lt;a href=&quot;http://www.amazon.com/Probabilistic-Recognition-Stochastic-Modelling-Probability/dp/0387946187&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;www.amazon.com&amp;#x2F;Probabilistic-Recognition-Stochastic-M...&lt;/a&gt;) have a really great picture of this in their book.</text></item></parent_chain><comment><author>tel</author><text>Totally agree, though I&amp;#x27;d call that maybe &amp;quot;picking the wrong shape&amp;quot; for your model space. You can pay a whole lot but if you cannot admit a shape that gets close to the truth then you&amp;#x27;re spending your data in vain.</text></comment>