<story><title>A ragtag band of internet friends became the best at forecasting world events</title><url>https://www.vox.com/future-perfect/2024/2/13/24070864/samotsvety-forecasting-superforecasters-tetlock</url></story><parent_chain><item><author>vlovich123</author><text>The base rate arguing seems like specious reasoning. For example, if you had a volcano that erupts roughly ever 100 years, base rate reasoning using the past 99 years of data would suggest that the probability is 0 and using 1000 years of data would suggest it&amp;#x27;s ~10% when in reality your base rate in the year following an eruption is 0 with every passing year your probability of an eruption would increase &amp;amp; increase past 10% for every year past 100 that goes without explosion. Same goes for something like war where pressures build up and war becomes more likely rather than less. So getting judged that you&amp;#x27;re better at predicting by giving low probabilities for rare events isn&amp;#x27;t that insightful because you&amp;#x27;d be outperformed by someone who predicts a black swan event because the magnitude of the event matters.&lt;p&gt;&amp;gt; The prediction got some press attention and earned rejoinders from nuclear experts like Peter Scoblic, who argued it significantly understated the risk of a nuclear exchange. It was a big moment for the group — but also an example of a prediction that’s very, very difficult to get right. The further you’re straying from the ordinary course of history (and a nuclear bomb going off in London would be straying very far), the harder this is.&lt;p&gt;Yup, the group got it right but predicting a rare event doesn&amp;#x27;t happen isn&amp;#x27;t that difficult, it&amp;#x27;s just notable because everyone was overly freaked out, particularly in the media due to self-repeated sensationalism. Peter Scoblic is correct that the risk is significantly understated because it&amp;#x27;s not correctly adjusting for the impact of the black swan event happening (e.g. if a nuclear explosion were to occur, you&amp;#x27;d expect nuclear retaliations).</text></item></parent_chain><comment><author>NunoSempere</author><text>&amp;gt; base rate reasoning using the past 99 years of data would suggest that the probability is 0&lt;p&gt;Looking at edge cases is good for sanity checking, so it&amp;#x27;s a good habit, and I commend you.&lt;p&gt;In your example, though, we can also consider the base rate of an event which hasn&amp;#x27;t happened in 99 years as 1&amp;#x2F;101, per Laplace&amp;#x27;s rule of succession. &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;en.m.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Rule_of_succession&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;en.m.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Rule_of_succession&lt;/a&gt;</text></comment>
<story><title>A ragtag band of internet friends became the best at forecasting world events</title><url>https://www.vox.com/future-perfect/2024/2/13/24070864/samotsvety-forecasting-superforecasters-tetlock</url></story><parent_chain><item><author>vlovich123</author><text>The base rate arguing seems like specious reasoning. For example, if you had a volcano that erupts roughly ever 100 years, base rate reasoning using the past 99 years of data would suggest that the probability is 0 and using 1000 years of data would suggest it&amp;#x27;s ~10% when in reality your base rate in the year following an eruption is 0 with every passing year your probability of an eruption would increase &amp;amp; increase past 10% for every year past 100 that goes without explosion. Same goes for something like war where pressures build up and war becomes more likely rather than less. So getting judged that you&amp;#x27;re better at predicting by giving low probabilities for rare events isn&amp;#x27;t that insightful because you&amp;#x27;d be outperformed by someone who predicts a black swan event because the magnitude of the event matters.&lt;p&gt;&amp;gt; The prediction got some press attention and earned rejoinders from nuclear experts like Peter Scoblic, who argued it significantly understated the risk of a nuclear exchange. It was a big moment for the group — but also an example of a prediction that’s very, very difficult to get right. The further you’re straying from the ordinary course of history (and a nuclear bomb going off in London would be straying very far), the harder this is.&lt;p&gt;Yup, the group got it right but predicting a rare event doesn&amp;#x27;t happen isn&amp;#x27;t that difficult, it&amp;#x27;s just notable because everyone was overly freaked out, particularly in the media due to self-repeated sensationalism. Peter Scoblic is correct that the risk is significantly understated because it&amp;#x27;s not correctly adjusting for the impact of the black swan event happening (e.g. if a nuclear explosion were to occur, you&amp;#x27;d expect nuclear retaliations).</text></item></parent_chain><comment><author>draaglom</author><text>I think the article focused on base rates because they&amp;#x27;re a relatively unusual and legible &amp;quot;trick&amp;quot; to coming up with a forecast, but really they&amp;#x27;re only one element of a forecast; typically a forecaster will think about many different ways to &amp;quot;attack&amp;quot; a question and synthesize them (somehow!). Choice of denominator for your base rate is very important also and can radically change the answer you get.&lt;p&gt;The sites which host these forecasting competitions correct for the bias against rare events through what&amp;#x27;s called &amp;quot;proper scoring&amp;quot; rules -- there&amp;#x27;s some specific maths to it, but the short version is that you&amp;#x27;re exponentially rewarded for being a correct contrarian and exponentially punished for being confidently wrong.&lt;p&gt;There are limits to that too, of course -- the folks in the article will &amp;quot;only&amp;quot; have made on the order of mid hundreds to low thousands of predictions, so roughly speaking, you can expect these people to be calibrated for 1% or 0.5% odds but probably not 0.1% odds.</text></comment>