<story><title>What&apos;s Really Warming the World?</title><url>http://www.bloomberg.com/graphics/2015-whats-warming-the-world/</url></story><parent_chain><item><author>wnewman</author><text>&amp;quot;The real trick is to see how well your model extrapolates from the data you have out into the future.&amp;quot;&lt;p&gt;That is the most common way to show the modeller is not shamelessly overfitting.:-| Another way, though, is less common but not vanishingly uncommon: the model may be so much simpler than the data it fits that overfitting is not a plausible explanation. (Roughly there are too many bits of entropy in the match to the data to have been packed into the model no matter how careless or dishonest you might have been about overfitting.) E.g., quantum mechanics is fundamentally pretty simple --- I can&amp;#x27;t quantify it exactly, but I think 5 pages of LaTeX output, in a sort of telegraphic elevator pitch cheat sheet style, would suffice to explain it to 1903 Einstein or Planck well enough that they could quickly figure out how to do calculations. Indeed, one page might suffice. And there are only a few adjustable parameters (particle&amp;#x2F;nucleus masses, Planck&amp;#x27;s constant, and less than a dozen others). And it matches sizable tables of spectroscopic data to more than six significant figures. (Though admittedly I dunno whether the non-hydrogen calculations would have been practical in 1903.) For the usual information-theoretical reasons, overfitting is not a real possibility: even if you don&amp;#x27;t check QM with spectroscopic measurements on previously unstudied substances, you can be pretty sure that QM is a good model. (Of course you still have to worry about it potentially breaking down in areas you haven&amp;#x27;t investigated yet, but at least it impressively captures regularities in the area you have investigated.)</text></item><item><author>jasonkester</author><text>Cool visualization.&lt;p&gt;It&amp;#x27;s worth keeping in mind that the modeled data lines up with reality because it&amp;#x27;s supposed to.  That&amp;#x27;s how you calibrate your model, by making sure it fits reality.&lt;p&gt;The real trick is to see how well your model extrapolates from the data you have out into the future.  As in, if you feed it data up to, say, 1990, will it correctly spit out 2015 temperatures that fit the reality of 2015, or will it spit out crazy 2015 predictions like the models that were built in 1990 did.  And, the bigger question: How will its predictions for 2040 (given 2015 data) match up to the reality over the next 25 years.&lt;p&gt;We seem to be getting a lot better at the modeling side.  That&amp;#x27;s a good thing, since the first couple decades of watching people panicking and fighting each other over whatever scary results came out of the first generation climate models wasn&amp;#x27;t any fun to watch.</text></item></parent_chain><comment><author>CWuestefeld</author><text>It&amp;#x27;s not just a question of how the model extrapolates from the input data itself. The actual input data may be in question as well, because there are always judgments involved in deciding how to measure, what &amp;quot;unreasonable&amp;quot; datapoints will be discarded, etc.&lt;p&gt;Read, for example, here:&lt;p&gt;&amp;quot;It is indisputable that a theory that is inconsistent with empirical data is a poor theory.  No theory should be accepted merely because of the beauty of its logic or because it leads to conclusions that are ideologically welcome or politically convenient.  Yet it is naive in the extreme to suppose that facts – especially the facts of the social sciences – speak for themselves.  Not only is it true that sound analysis is unavoidably a judgment-laden mix of rigorous reasoning (“theory”) with careful observation of the facts; it is also true that the facts themselves are in large part the product of theorizing. ...&amp;quot;&lt;p&gt;&lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;cafehayek.com&amp;#x2F;2015&amp;#x2F;04&amp;#x2F;theorizing-about-the-facts-theres-no-escaping-theory.html&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;cafehayek.com&amp;#x2F;2015&amp;#x2F;04&amp;#x2F;theorizing-about-the-facts-ther...&lt;/a&gt;</text></comment>
<story><title>What&apos;s Really Warming the World?</title><url>http://www.bloomberg.com/graphics/2015-whats-warming-the-world/</url></story><parent_chain><item><author>wnewman</author><text>&amp;quot;The real trick is to see how well your model extrapolates from the data you have out into the future.&amp;quot;&lt;p&gt;That is the most common way to show the modeller is not shamelessly overfitting.:-| Another way, though, is less common but not vanishingly uncommon: the model may be so much simpler than the data it fits that overfitting is not a plausible explanation. (Roughly there are too many bits of entropy in the match to the data to have been packed into the model no matter how careless or dishonest you might have been about overfitting.) E.g., quantum mechanics is fundamentally pretty simple --- I can&amp;#x27;t quantify it exactly, but I think 5 pages of LaTeX output, in a sort of telegraphic elevator pitch cheat sheet style, would suffice to explain it to 1903 Einstein or Planck well enough that they could quickly figure out how to do calculations. Indeed, one page might suffice. And there are only a few adjustable parameters (particle&amp;#x2F;nucleus masses, Planck&amp;#x27;s constant, and less than a dozen others). And it matches sizable tables of spectroscopic data to more than six significant figures. (Though admittedly I dunno whether the non-hydrogen calculations would have been practical in 1903.) For the usual information-theoretical reasons, overfitting is not a real possibility: even if you don&amp;#x27;t check QM with spectroscopic measurements on previously unstudied substances, you can be pretty sure that QM is a good model. (Of course you still have to worry about it potentially breaking down in areas you haven&amp;#x27;t investigated yet, but at least it impressively captures regularities in the area you have investigated.)</text></item><item><author>jasonkester</author><text>Cool visualization.&lt;p&gt;It&amp;#x27;s worth keeping in mind that the modeled data lines up with reality because it&amp;#x27;s supposed to.  That&amp;#x27;s how you calibrate your model, by making sure it fits reality.&lt;p&gt;The real trick is to see how well your model extrapolates from the data you have out into the future.  As in, if you feed it data up to, say, 1990, will it correctly spit out 2015 temperatures that fit the reality of 2015, or will it spit out crazy 2015 predictions like the models that were built in 1990 did.  And, the bigger question: How will its predictions for 2040 (given 2015 data) match up to the reality over the next 25 years.&lt;p&gt;We seem to be getting a lot better at the modeling side.  That&amp;#x27;s a good thing, since the first couple decades of watching people panicking and fighting each other over whatever scary results came out of the first generation climate models wasn&amp;#x27;t any fun to watch.</text></item></parent_chain><comment><author>darkmighty</author><text>While the general gist of your argument is right, I think there are some non-trivial ways to overfit. There are some 25 constants in the standard model apparently that describe  the world around us to enormous precision. This is so little information that of course the trivial &amp;#x27;overfitting by encoding observations directly&amp;#x27; will fail, but we could still be overfitting by having an excess number of variables: perhaps there&amp;#x27;s really some mechanism in neutrino physics that explains neutrino oscillation without needing some constants to describe how it really happens. This might in turn boost tremendously our predictive precision for neutrino oscillation to match the precision of the other more fundamental variables in the model, for example. But I think you&amp;#x27;re right that it&amp;#x27;s so little data that we have some strong information theoretic guarantees that at least the model will have predictive power matching the precision of previous measurements.</text></comment>