<commit_before><!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
<html>
    <head>
        <title>An Educated Guess</title>
        <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"/>
        <link rel="stylesheet" media="screen" href="css/bayesian.css">
        <link rel="stylesheet" media="screen" href="css/chart.css">
        <link rel="stylesheet" href="http://code.jquery.com/ui/1.10.3/themes/smoothness/jquery-ui.css">
        <link rel="stylesheet" media="screen" href="css/jquery.qtip.css">
    </head>
    <body>
        <div id="info">
            <h1 id="name">an educated guess</h1>
            <p>
            A tool for calibrated probability estimation
            </p>
            
            <div>
                This application was designed for <a href="http://chrome.google.com">Google Chrome</a> and may not function properly in other browsers.
            </div>

            <div>
                <h2>Recent Changes</h2>
                <table width="100%" cellpadding="0" cellspacing="0">

                    <tr class="visited">
                        <td valign="top">2013 Aug 10</td>
                        <td valign="top">GAE deployment</td>
                    </tr>
                    
                    <tr class="visited">
                        <td valign="top">2013 Aug 08</td>
                        <td valign="top">Device support</td>
                    </tr>

                    <tr class="visited">
                        <td valign="top">2013 July 31</td>
                        <td valign="top">Started project</td>
                    </tr>
                </table>
            </div>
        </div>

        <div id="main">
            <h1>
                about &#160;&#160;&#160;
                <a class="nav" href="calibrate.html">calibrate</a>&#160;&#160;&#160;
                <a class="nav" href="think.html">think probabilistically</a>&#160;&#160;&#160;
            </h1>

            <div class="text">
                <table width="100%" cellpadding="0" cellspacing="0">
                    <tr>
                        <td valign="top" style="padding-right: 25px;">
                        <h6 style="padding-top: 10px;">the bad news</h6>

                        <p>
                        You are a terrible guesser. 
                        </p>

                        <p>
                        I'm also a terrible guesser. As are most people, with the exception of some well-trained meteorologists and professional gamblers<sup class="cite" title="LichtensteinEtAl1982"></sup><sup class="cite" title="Silver2012"></sup>.
                        </p>

                        <p>
                        'Guessing' here refers to a properly calibrated probability estimate. If you were to guess the gross domestic earnings of <em>The Goonies</em>, your answer would probably be wrong, not because of profound personal faults, but because the question tests exact domain knowledge. Rephrasing the question as:
                        <blockquote>
                        &ldquo;What are your high and low estimates for the gross domestic earnings of the <em>The Goonies</em>? What interval estimate of <a href="https://en.wikipedia.org/wiki/Confidence_interval">confidence</a> would you give this range?&rdquo;
                        </blockquote>

                        The answer might be:
                        <blockquote>
                            &ldquo;Between $100,000 to $100,000,000, inclusive, with 90% confidence.&rdquo;<sup class="note" title="The exact <a href='http://www.boxofficemojo.com/movies/?id=goonies.htm'>answer</a> is $61,389,680.">[note]</sup>
                        </blockquote>

                        Given enough data points in a sufficiently ideal world, the estimated confidence interval would match the observed accuracy. In other words, the range estimates given with 90% confidence should &ndash; on average and with minimal variance &ndash; be correct 90% of the time<sup class="cite" title="AlpertRaffia1982"></sup>. This changes the exercise from a measurement of trivia knowledge to a measurement of ability to gauge uncertainty.  However, much fewer than 90% of answers with 90% confidence will contain the actual value because humans habitually display overconfident estimates. What we know is about 10-35 percentage points less than what we think we know in this example<sup class="cite" title="LichtensteinEtAl1982"></sup>.
                        </p>

                        <h6 style="padding-top: 10px;">cognitive bugs</h6>

                        <p>
                        Plenty of empirical research explores the limits and prejudices of probability estimates, and a few present theories on the potential causes, most of which can be found among other lists of cognitive biases<sup class="cite" title="Wilson1994"></sup>.
                        <ul>
                        <li><a href="http://en.wikipedia.org/wiki/Anchoring">Anchoring</a>: the wording and structure of the questions can decrease epistemic accuracy by up to 53 percentage points<sup class="cite" title="LichtensteinEtAl1982"></sup>.</li>
                        <li>Pattern-seeking habits of humans create the illusion of signal where there is only noisy data<sup class="cite" title="Silver2012"> 12.</sup>.</li>
                        <li><a href="http://www.princeton.edu/~kahneman/docs/Publications/prospect_theory.pdf">Prospect theory</a>: small probabilities are overweighed, especially when attached to high-consequence events<sup class="cite" title="Plous1993"> 100-101.</sup>.</li>
                        <li>Our educational training emphasizes algebra and calculus as the end goals, <a href="http://www.ted.com/talks/arthur_benjamin_s_formula_for_changing_math_education.html">not probability and statistics</a>.</li>
                        <li>We <a href="http://en.wikipedia.org/wiki/Base_rate_neglect">ignore</a> prior probabilities.</li>
                        <li>Many social and professional systems reward overconfidence<sup class="cite" title="RadzevickMoore2009"></sup>.</li>
                        <li>In situations determined primarily by chance, we often build narratives<sup class="note" title="The term 'Narrative Fallacy' originates from Taleb:
                        <blockquote><p>
&ldquo;The narrative fallacy addresses our limited ability to look at sequences of facts without weaving an explanation into them, or, equivalently, forcing a logical link, <i>an arrow of relationship</i> upon them. Explanations bind facts together. They make them all the more easily remembered; they help them <i>make more sense</i>. Where this propensity can go wrong is when it increases our <i>impression</i> of understanding.&rdquo;
                            </p></blockquote>
                            <p> &mdash;<NAME>, <i>The Black Swan</i> (p63-4)</p>">[note]</sup> to coherently explain the events, giving the illusion of control<sup class="cite" title="Kahneman2011"></sup>. E.g., market fluctuations due to labor reports, portfolio performance due to investment strategies or combat effectiveness predicted by training exercises<sup class="note" title="Narrative building is also used in the benefit of data science: to minimize the effect of 'overfitting', or forcing a quantitative prediction model to prior data (Silver2012, p196).">[note]</sup>.</li>
                        </ul>

                        These biases are to estimation as <a href="http://en.wikipedia.org/wiki/Checker_shadow_illusion">geometrical-optical illusions</a> are to psychometrics. A simple change of the problem context causes a predictable change in the perceived reality. In general, humans have a very troubled relationship with uncertainty. We don't understand it instinctually, we don't communicate it well<sup class="cite" title="Marx2013"></sup> and we're willing to pay<sup class="cite" title="Knight1921"> <a href="http://www.econlib.org/library/Knight/knRUP1.html#Pt.I,Ch.II">Part I, Chapter II</a>.</sup><sup class="cite" title="Plous1993"> 100-101.</sup> to avoid it.
                        </p>


                        <h6 style="padding-top: 10px;">you should care</h6>

                        <p>
                        Even if you don't live in a region with legalized gambling or work in a forecasting profession, everyday failures of estimation hurt your quality of life, whether due to inaccurate project estimates, poor investments or being late to your next appointment. You make decisions based on uncertainty and imperfect knowledge, knowing much less than you think you know. As far as ubiquitous problems of human existence, it's right up there with communicable disease<sup class="note" title="<blockquote>&ldquo;No problem in judgement in decision making is more prevalent and more potentially catastrophic than overconfidence. As loving Janis (1982) documented in his work on groupthink, American overconfidence enabled the Japanese to destroy Pearl Harbor in World War II. Overconfidence also played a role in the disastrous decision to launch the U.S. space shuttle Challenger. Before the shuttle exploded on its twenty-fifth mission, NASA's official launch risk estimate was 1 catastrophic failure in 100,000 launches (Feynman, 1988, February). This risk estimate is roughly equivalent to launching the shuttle once per day and expecting to see only one accident in three centuries.&rdquo;</blockquote> <p> &mdash;<NAME>, <i>The Psychology of Judgment and Decision Making</i> (p217)</p>">[note]</sup>.
                        </p>
                        <p>
                        More importantly, inability to accurately estimate closes the door to powerful tools of probabilistic thinking<sup class="cite" title="Jeffery2002"></sup>. With accurate prior probabilities, Bayesian prediction<sup class="note" title="Bayesian is sometimes called 'subjectivist'.">[note]</sup> avoids the nuances of frequentist statistics, while allowing your mental model to adapt as the facts change. It's something for which the Army<sup class="cite" title="LichtensteinFischhoff1978"></sup> and Air Force<sup class="cite" title="GunzelmannGluck2004"></sup> train, and M.D.s understand through years of experience<sup class="cite" title="Gill2005"></sup><sup class="cite" title="LindeyEtAl1979"></sup>. Along with the <a href="http://en.wikipedia.org/wiki/Speed#Definition">distance-rate-time</a> equation, <a href="http://en.wikipedia.org/wiki/Time_value_of_money#Formula">time-value</a> equation and <a href="http://en.wikipedia.org/wiki/Logical_equality#Alternative_descriptions">logical equalities</a>, <a href="http://en.wikipedia.org/wiki/Bayes_theorem">Bayes' Theorem</a> is one of the those unreasonably effective structures of math, which internalizing will vastly improve your thinking<sup class="cite" title="McIntyre2007"></sup>.
                        </p>

                        <h6 style="padding-top: 10px;">mensa mea bona est</h6>

                        Overconfidence follows a predictable pattern. It is usual for difficult assessments (although slightly less for true/false tests<sup class="cite" title="Hubbard2010"> p64.</sup>). In some cases, very easy questions inspire underconfidence<sup class="cite" title="LichtensteinEtAl1982"></sup>. Calibration techniques can help to correct this:
                        <ul>
                            <li>Consider the reasons why your judgment might be wrong<sup class="cite" title="Plous1993"> p228.</sup>.</li>
                            <li>Range estimates can reduce the anchoring effect of a point estimate, particularly by working towards a narrow range from an absurdly large range<sup class="cite" title="Hubbard2010"> p64-5.</sup>.</li>
                        </ul>
                        Things that don't fix overconfidence:
                        <ul>
                            <li>More information. Paradoxically, providing more information to the problem increases one's confidence in the answer, but not accuracy<sup class="cite" title="KassinFong1999"></sup><sup class="cite" title="Oskamp1965"></sup>.</li>
                        </ul>
 
                        <p>
                        Most importantly, feedback and iterative practice allow us to improve our estimation techniques<sup class="cite" title="LichtensteinEtAl1982"></sup>, hence the purpose of this project. Select the number of questions you want and the quiz will give you instant feedback on your progress.
                        </p>
                        
                        <p align="center">
                            <img alt="probability distribution" src="images/distribution_gradients.png">
                        </p>
                        
                        <p>
                        The darker the color, the more questions have been answered at that confidence, and the more reliable the results will be.
                        </p>
                        <p>
                        When choosing your confidence level, 50% confidence indicates that you have no idea which answer is correct. 100% indicates absolute certainty of the correct answer. The more questions you answer, the more accurate your calibration will be.
                        </p>

                        <p align="center">
                            <a class="button" href="calibrate.html?n=20" title="about 4 minutes">20 questions</a>                        
                            <a class="button" href="calibrate.html?n=40" title="about 8 minutes">40 questions</a>
                            <a class="button" href="calibrate.html?n=70" title="about 14 minutes">70 questions</a>
                            <a class="button" href="calibrate.html?n=100" title="about 25 minutes">100 questions</a>
                        </p>

                        <h6 style="padding-top: 10px;">other examples</h6>
                            If you want to try other types of calibrated probability assessments:

                        <ul>
                            <li>A <a href="http://lesswrong.com/lw/1f8/test_your_calibration/">list</a> of them</li>
                            <li><a href="http://www.acceleratingfuture.com/steven/?p=136">Aumann Game</a>: a manually scoring example</li>
                            <li>An <a href="http://calibratedprobabilityassessment.org">automated quiz</a> that produces a <a href="http://calibratedprobabilityassessment.org/graph.php?y=0.28571428571429-0.90909090909091-0.75-1-1&x=55-65-75-85-95">nifty graph</a>.</li>
                            <li>A <a href="http://messymatters.com/calibration/">range estimate quiz</a> reproduced from <a href="http://www.amazon.com/gp/product/0671726099/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0671726099&linkCode=as2&tag=sethrylan-20"><em>Decision Traps</em></a>, and also in Plous<sup class="cite" title="Plous1993"></sup>.</li>
                        </ul> 

                        <h6 style="padding-top: 10px;">sources</h6>

<div class="csl-entry" id="AlpertRaffia1982"><NAME>, and <NAME>. <a href="http://faculty.washington.edu/jmiyamot/p466/alpertm%20prog%20report%20on%20training%20o%20prob%20assessors.pdf">A Progress Report on the Training of Probability Assessors.</a> In <a href="http://www.amazon.com/gp/product/0521284147/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0521284147&linkCode=as2&tag=sethrylan-20"><i>Judgment Under Uncertainty: Heuristics and Biases</i></a>, edited by <NAME>, <NAME>, and <NAME>, 294305. Cambridge University Press, 1982. <a href="http://dx.doi.org/10.1017/CBO9780511809477.022">http://dx.doi.org/10.1017/CBO9780511809477.022</a>.</div>

<div class="csl-entry" id="Gill2005"><NAME>. <a href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC557240/pdf/bmj33001080.pdf">Why Clinicians Are Natural Bayesians.</a> <i>BMJ</i> 330, no. 7499 (May 7, 2005): 10801083. doi:10.1136/bmj.330.7499.1080.</div>
  
<div class="csl-entry" id="GunzelmannGluck2004"><NAME>., and <NAME>. <a href="http://act-r.psy.cmu.edu/papers/710/gunzelmann_gluck-2004.pdf">Knowledge Tracing for Complex Training Applications: Beyond Bayesian Mastery Estimates.</a> In <i>Proceedings of the Thirteenth Conference on Behavior Representation in Modeling and Simulation</i>, 383384. Orlando, FL: Simulation Interoperability Standards Organization, 2004.</div>

<div class="csl-entry" id="Hubbard2010"><NAME>. <a href="http://www.amazon.com/gp/product/0470539399/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0470539399&linkCode=as2&tag=sethrylan-20"><!--http://www.jpmeloche.com/crr/ebooksclub.org__How_to_Measure_Anything__Finding_the_Value_of_Intangibles_in_Business__Second_Edition.pdf--><i>How to Measure Anything Finding the Value of Intangibles in Business</i></a>. 2ed. Wiley, 2010.</div>

<div class="csl-entry" id="Jeffery2002"><NAME>. <a href="http://www.princeton.edu/~bayesway/Book*.pdf"><i>Subjective Probability: The Real Thing</i></a>. Cambridge University Press, 2002.</div>
  
<div class="csl-entry" id="Kahneman2011"><NAME>. <a href="http://www.nytimes.com/2011/10/23/magazine/dont-blink-the-hazards-of-confidence.html">Dont Blink! The Hazards of Confidence.</a> <i>The New York Times</i>, October 19, 2011, sec. Magazine.</div>
  
<div class="csl-entry" id="KassinFong1999"><NAME>., and <NAME>. <a href="http://web.williams.edu/Psychology/Faculty/Kassin/files/kassin_fong_1999.pdf">Im Innocent!: Effects of Training on Judgments of Truth and Deception in the Interrogation Room.</a> <i>Law and Human Behavior</i> 23, no. 5 (October 1, 1999): 499516. doi:10.1023/A:1022330011811.</div>

<div class="csl-entry" id="Knight1921"><NAME>. (<NAME>). <a href="http://www.amazon.com/gp/product/1602060053/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=1602060053&linkCode=as2&tag=sethrylan-20"><!--http://www.econlib.org/library/Knight/knRUP.html--><i>Risk, Uncertainty and Profit</i></a>. Boston, New York, Houghton Mifflin Company, 1921.</div>

<div class="csl-entry" id="LichtensteinFischhoff1978">Lichtenstein, Sarah, and <NAME>. <a href="http://www.dtic.mil/dtic/tr/fulltext/u2/a069703.pdf"><i>Training for Calibration</i></a>, November 1978.</div>
  
<div class="csl-entry" id="LichtensteinEtAl1982">Lichtenstein, Sarah, <NAME>, and <NAME>. <a href="http://www.dtic.mil/dtic/tr/fulltext/u2/a101986.pdf">Calibration of Probabilities: The State of the Art to 1980.</a> In <a href="http://www.amazon.com/gp/product/0521284147/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0521284147&linkCode=as2&tag=sethrylan-20"><i>Judgment Under Uncertainty: Heuristics and Biases</i></a>, edited by <NAME>, <NAME>, and <NAME>, 306334. Cambridge, UK: Cambridge University Press, 1982.</div>

<div class="csl-entry" id="LindeyEtAl1979"><NAME>., <NAME>, and <NAME>. <a href="citations/Lindly_et_al-On_the_Reconciliation_of_Probability_Assessments.pdf">On the Reconciliation of Probability Assessments.</a> <i>Journal of the Royal Statistical Society. Series A (General)</i> 142, no. 2 (January 1, 1979): 146180. doi:10.2307/2345078.</div>

<div class="csl-entry" id="Marx2013"><NAME>.  <a href="http://www.nature.com/nmeth/journal/v10/n7/full/nmeth.2530.html">Data Visualization: Ambiguity as a Fellow Traveler.</a> <i>Nature Methods</i> 10, no. 7 (July 2013): 613615. doi:10.1038/nmeth.2530.</div>
  
<div class="csl-entry" id="McIntyre2007"><NAME>. <a href="http://www.atm.damtp.cam.ac.uk/mcintyre/mcintyre-thinking-probabilistically.pdf">On Thinking Probabilistically.</a> In <i>Extreme Events (Proc. 15th Aha Hulikoa Workshop)</i>, 153161. U. of Hawaii: SOEST, 2007.</div>
 
<div class="csl-entry" id="Oskamp1965"><NAME>. <a href="citations/Oskamp-Overconfidence_in_Case_Study_Judgements.pdf">Overconfidence in Case-study Judgments.</a> <i>Journal of Consulting Psychology</i> 29, no. 3 (1965): 261265. doi:10.1037/h0022125.</div>

<div class="csl-entry" id="Plous1993"><NAME>. <a href="http://www.amazon.com/gp/product/0070504776/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0070504776&linkCode=as2&tag=sethrylan-20"><i>The Psychology of Judgment and Decision Making</i></a>. New York: McGraw-Hill, 1993.</div>

<div class="csl-entry" id="RadzevickMoore2009"><NAME>., and <NAME>. <a href="http://www.gsb.stanford.edu/sites/default/files/documents/ob_01_09_moore.pdf">Competing to Be Certain (but Wrong): Social Pressure and Overprecision in Judgment.</a> <i>Academy of Management Proceedings</i> 2009, no. 1 (August 1, 2009): 16. doi:10.5465/AMBPP.2009.44246308.</div>

<div class="csl-entry" id="Silver2012"><NAME>. <a href="http://www.amazon.com/gp/product/159420411X/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=159420411X&linkCode=as2&tag=sethrylan-20"><i>The Signal and the Noise: Why So Many Predictions Fail  but Some Dont</i></a>. 1ed. Penguin Press HC, The, 2012.</div>

<div class="csl-entry" id="Wilson1994"><NAME>. <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.71.4909&rep=rep1&type=pdf">Cognitive Factors Affecting Subjective Probability Assessment,</a> 1994.</div>

                        </td>
                        <td width="20%" valign="top">
                        </td>
                    </tr>
                </table>
            </div>

        </div>
        <script src="//ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
        <script src="//ajax.googleapis.com/ajax/libs/jqueryui/1.10.3/jquery-ui.min.js"></script>        
        <script src="js/jquery.ui.touch-punch.min.js"></script>
        <script src="js/jquery.qtip.js"></script><commit_msg>Changed smart-quotes to simple quotes.<commit_after><!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
<html>
    <head>
        <title>An Educated Guess</title>
        <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"/>
        <link rel="stylesheet" media="screen" href="css/bayesian.css">
        <link rel="stylesheet" media="screen" href="css/chart.css">
        <link rel="stylesheet" href="http://code.jquery.com/ui/1.10.3/themes/smoothness/jquery-ui.css">
        <link rel="stylesheet" media="screen" href="css/jquery.qtip.css">
    </head>
    <body>
        <div id="info">
            <h1 id="name">an educated guess</h1>
            <p>
            A tool for calibrated probability estimation
            </p>
            
            <div>
                This application was designed for <a href="http://chrome.google.com">Google Chrome</a> and may not function properly in other browsers.
            </div>

            <div>
                <h2>Recent Changes</h2>
                <table width="100%" cellpadding="0" cellspacing="0">

                    <tr class="visited">
                        <td valign="top">2013 Aug 10</td>
                        <td valign="top">GAE deployment</td>
                    </tr>
                    
                    <tr class="visited">
                        <td valign="top">2013 Aug 08</td>
                        <td valign="top">Device support</td>
                    </tr>

                    <tr class="visited">
                        <td valign="top">2013 July 31</td>
                        <td valign="top">Started project</td>
                    </tr>
                </table>
            </div>
        </div>

        <div id="main">
            <h1>
                about &#160;&#160;&#160;
                <a class="nav" href="calibrate.html">calibrate</a>&#160;&#160;&#160;
                <a class="nav" href="think.html">think probabilistically</a>&#160;&#160;&#160;
            </h1>

            <div class="text">
                <table width="100%" cellpadding="0" cellspacing="0">
                    <tr>
                        <td valign="top" style="padding-right: 25px;">
                        <h6 style="padding-top: 10px;">the bad news</h6>

                        <p>
                        You are a terrible guesser. 
                        </p>

                        <p>
                        I'm also a terrible guesser. As are most people, with the exception of some well-trained meteorologists and professional gamblers<sup class="cite" title="LichtensteinEtAl1982"></sup><sup class="cite" title="Silver2012"></sup>.
                        </p>

                        <p>
                        'Guessing' here refers to a properly calibrated probability estimate. If you were to guess the gross domestic earnings of <em>The Goonies</em>, your answer would probably be wrong, not because of profound personal faults, but because the question tests exact domain knowledge. Rephrasing the question as:
                        <blockquote>
                        &ldquo;What are your high and low estimates for the gross domestic earnings of the <em>The Goonies</em>? What interval estimate of <a href="https://en.wikipedia.org/wiki/Confidence_interval">confidence</a> would you give this range?&rdquo;
                        </blockquote>

                        The answer might be:
                        <blockquote>
                            &ldquo;Between $100,000 to $100,000,000, inclusive, with 90% confidence.&rdquo;<sup class="note" title="The exact <a href='http://www.boxofficemojo.com/movies/?id=goonies.htm'>answer</a> is $61,389,680.">[note]</sup>
                        </blockquote>

                        Given enough data points in a sufficiently ideal world, the estimated confidence interval would match the observed accuracy. In other words, the range estimates given with 90% confidence should &ndash; on average and with minimal variance &ndash; be correct 90% of the time<sup class="cite" title="AlpertRaffia1982"></sup>. This changes the exercise from a measurement of trivia knowledge to a measurement of ability to gauge uncertainty.  However, much fewer than 90% of answers with 90% confidence will contain the actual value because humans habitually display overconfident estimates. What we know is about 10-35 percentage points less than what we think we know in this example<sup class="cite" title="LichtensteinEtAl1982"></sup>.
                        </p>

                        <h6 style="padding-top: 10px;">cognitive bugs</h6>

                        <p>
                        Plenty of empirical research explores the limits and prejudices of probability estimates, and a few present theories on the potential causes, most of which can be found among other lists of cognitive biases<sup class="cite" title="Wilson1994"></sup>.
                        <ul>
                        <li><a href="http://en.wikipedia.org/wiki/Anchoring">Anchoring</a>: the wording and structure of the questions can decrease epistemic accuracy by up to 53 percentage points<sup class="cite" title="LichtensteinEtAl1982"></sup>.</li>
                        <li>Pattern-seeking habits of humans create the illusion of signal where there is only noisy data<sup class="cite" title="Silver2012"> 12.</sup>.</li>
                        <li><a href="http://www.princeton.edu/~kahneman/docs/Publications/prospect_theory.pdf">Prospect theory</a>: small probabilities are overweighed, especially when attached to high-consequence events<sup class="cite" title="Plous1993"> 100-101.</sup>.</li>
                        <li>Our educational training emphasizes algebra and calculus as the end goals, <a href="http://www.ted.com/talks/arthur_benjamin_s_formula_for_changing_math_education.html">not probability and statistics</a>.</li>
                        <li>We <a href="http://en.wikipedia.org/wiki/Base_rate_neglect">ignore</a> prior probabilities.</li>
                        <li>Many social and professional systems reward overconfidence<sup class="cite" title="RadzevickMoore2009"></sup>.</li>
                        <li>In situations determined primarily by chance, we often build narratives<sup class="note" title="The term 'Narrative Fallacy' originates from Taleb:
                        <blockquote><p>
&ldquo;The narrative fallacy addresses our limited ability to look at sequences of facts without weaving an explanation into them, or, equivalently, forcing a logical link, <i>an arrow of relationship</i> upon them. Explanations bind facts together. They make them all the more easily remembered; they help them <i>make more sense</i>. Where this propensity can go wrong is when it increases our <i>impression</i> of understanding.&rdquo;
                            </p></blockquote>
                            <p> &mdash;<NAME>, <i>The Black Swan</i> (p63-4)</p>">[note]</sup> to coherently explain the events, giving the illusion of control<sup class="cite" title="Kahneman2011"></sup>. E.g., market fluctuations due to labor reports, portfolio performance due to investment strategies or combat effectiveness predicted by training exercises<sup class="note" title="Narrative building is also used in the benefit of data science: to minimize the effect of 'overfitting', or forcing a quantitative prediction model to prior data (Silver2012, p196).">[note]</sup>.</li>
                        </ul>

                        These biases are to estimation as <a href="http://en.wikipedia.org/wiki/Checker_shadow_illusion">geometrical-optical illusions</a> are to psychometrics. A simple change of the problem context causes a predictable change in the perceived reality. In general, humans have a very troubled relationship with uncertainty. We don't understand it instinctually, we don't communicate it well<sup class="cite" title="Marx2013"></sup> and we're willing to pay<sup class="cite" title="Knight1921"> <a href="http://www.econlib.org/library/Knight/knRUP1.html#Pt.I,Ch.II">Part I, Chapter II</a>.</sup><sup class="cite" title="Plous1993"> 100-101.</sup> to avoid it.
                        </p>


                        <h6 style="padding-top: 10px;">you should care</h6>

                        <p>
                        Even if you don't live in a region with legalized gambling or work in a forecasting profession, everyday failures of estimation hurt your quality of life, whether due to inaccurate project estimates, poor investments or being late to your next appointment. You make decisions based on uncertainty and imperfect knowledge, knowing much less than you think you know. As far as ubiquitous problems of human existence, it's right up there with communicable disease<sup class="note" title="<blockquote>&ldquo;No problem in judgement in decision making is more prevalent and more potentially catastrophic than overconfidence. As loving Janis (1982) documented in his work on groupthink, American overconfidence enabled the Japanese to destroy Pearl Harbor in World War II. Overconfidence also played a role in the disastrous decision to launch the U.S. space shuttle Challenger. Before the shuttle exploded on its twenty-fifth mission, NASA's official launch risk estimate was 1 catastrophic failure in 100,000 launches (Feynman, 1988, February). This risk estimate is roughly equivalent to launching the shuttle once per day and expecting to see only one accident in three centuries.&rdquo;</blockquote> <p> &mdash;<NAME>, <i>The Psychology of Judgment and Decision Making</i> (p217)</p>">[note]</sup>.
                        </p>
                        <p>
                        More importantly, inability to accurately estimate closes the door to powerful tools of probabilistic thinking<sup class="cite" title="Jeffery2002"></sup>. With accurate prior probabilities, Bayesian prediction<sup class="note" title="Bayesian is sometimes called 'subjectivist'.">[note]</sup> avoids the nuances of frequentist statistics, while allowing your mental model to adapt as the facts change. It's something for which the Army<sup class="cite" title="LichtensteinFischhoff1978"></sup> and Air Force<sup class="cite" title="GunzelmannGluck2004"></sup> train, and M.D.s understand through years of experience<sup class="cite" title="Gill2005"></sup><sup class="cite" title="LindeyEtAl1979"></sup>. Along with the <a href="http://en.wikipedia.org/wiki/Speed#Definition">distance-rate-time</a> equation, <a href="http://en.wikipedia.org/wiki/Time_value_of_money#Formula">time-value</a> equation and <a href="http://en.wikipedia.org/wiki/Logical_equality#Alternative_descriptions">logical equalities</a>, <a href="http://en.wikipedia.org/wiki/Bayes_theorem">Bayes' Theorem</a> is one of the those unreasonably effective structures of math, which internalizing will vastly improve your thinking<sup class="cite" title="McIntyre2007"></sup>.
                        </p>

                        <h6 style="padding-top: 10px;">mensa mea bona est</h6>

                        Overconfidence follows a predictable pattern. It is usual for difficult assessments (although slightly less for true/false tests<sup class="cite" title="Hubbard2010"> p64.</sup>). In some cases, very easy questions inspire underconfidence<sup class="cite" title="LichtensteinEtAl1982"></sup>. Calibration techniques can help to correct this:
                        <ul>
                            <li>Consider the reasons why your judgment might be wrong<sup class="cite" title="Plous1993"> p228.</sup>.</li>
                            <li>Range estimates can reduce the anchoring effect of a point estimate, particularly by working towards a narrow range from an absurdly large range<sup class="cite" title="Hubbard2010"> p64-5.</sup>.</li>
                        </ul>
                        Things that don't fix overconfidence:
                        <ul>
                            <li>More information. Paradoxically, providing more information to the problem increases one's confidence in the answer, but not accuracy<sup class="cite" title="KassinFong1999"></sup><sup class="cite" title="Oskamp1965"></sup>.</li>
                        </ul>
 
                        <p>
                        Most importantly, feedback and iterative practice allow us to improve our estimation techniques<sup class="cite" title="LichtensteinEtAl1982"></sup>, hence the purpose of this project. Select the number of questions you want and the quiz will give you instant feedback on your progress.
                        </p>
                        
                        <p align="center">
                            <img alt="probability distribution" src="images/distribution_gradients.png">
                        </p>
                        
                        <p>
                        The darker the color, the more questions have been answered at that confidence, and the more reliable the results will be.
                        </p>
                        <p>
                        When choosing your confidence level, 50% confidence indicates that you have no idea which answer is correct. 100% indicates absolute certainty of the correct answer. The more questions you answer, the more accurate your calibration will be.
                        </p>

                        <p align="center">
                            <a class="button" href="calibrate.html?n=20" title="about 4 minutes">20 questions</a>                        
                            <a class="button" href="calibrate.html?n=40" title="about 8 minutes">40 questions</a>
                            <a class="button" href="calibrate.html?n=70" title="about 14 minutes">70 questions</a>
                            <a class="button" href="calibrate.html?n=100" title="about 25 minutes">100 questions</a>
                        </p>

                        <h6 style="padding-top: 10px;">other examples</h6>
                            If you want to try other types of calibrated probability assessments:

                        <ul>
                            <li>A <a href="http://lesswrong.com/lw/1f8/test_your_calibration/">list</a> of them</li>
                            <li><a href="http://www.acceleratingfuture.com/steven/?p=136">Aumann Game</a>: a manually scoring example</li>
                            <li>An <a href="http://calibratedprobabilityassessment.org">automated quiz</a> that produces a <a href="http://calibratedprobabilityassessment.org/graph.php?y=0.28571428571429-0.90909090909091-0.75-1-1&x=55-65-75-85-95">nifty graph</a>.</li>
                            <li>A <a href="http://messymatters.com/calibration/">range estimate quiz</a> reproduced from <a href="http://www.amazon.com/gp/product/0671726099/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0671726099&linkCode=as2&tag=sethrylan-20"><em>Decision Traps</em></a>, and also in Plous<sup class="cite" title="Plous1993"></sup>.</li>
                        </ul> 

                        <h6 style="padding-top: 10px;">sources</h6>

<div class="csl-entry" id="AlpertRaffia1982"><NAME>, and <NAME>. <a href="http://faculty.washington.edu/jmiyamot/p466/alpertm%20prog%20report%20on%20training%20o%20prob%20assessors.pdf">"A Progress Report on the Training of Probability Assessors."</a> In <a href="http://www.amazon.com/gp/product/0521284147/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0521284147&linkCode=as2&tag=sethrylan-20"><i>Judgment Under Uncertainty: Heuristics and Biases</i></a>, edited by <NAME>, <NAME>, and <NAME>, 294305. Cambridge University Press, 1982. <a href="http://dx.doi.org/10.1017/CBO9780511809477.022">http://dx.doi.org/10.1017/CBO9780511809477.022</a>.</div>

<div class="csl-entry" id="Gill2005"><NAME>. <a href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC557240/pdf/bmj33001080.pdf">"Why Clinicians Are Natural Bayesians."</a> <i>BMJ</i> 330, no. 7499 (May 7, 2005): 10801083. doi:10.1136/bmj.330.7499.1080.</div>
  
<div class="csl-entry" id="GunzelmannGluck2004"><NAME>., and <NAME>. <a href="http://act-r.psy.cmu.edu/papers/710/gunzelmann_gluck-2004.pdf">"Knowledge Tracing for Complex Training Applications: Beyond Bayesian Mastery Estimates."</a> In <i>Proceedings of the Thirteenth Conference on Behavior Representation in Modeling and Simulation</i>, 383384. Orlando, FL: Simulation Interoperability Standards Organization, 2004.</div>

<div class="csl-entry" id="Hubbard2010"><NAME>. <a href="http://www.amazon.com/gp/product/0470539399/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0470539399&linkCode=as2&tag=sethrylan-20"><!--http://www.jpmeloche.com/crr/ebooksclub.org__How_to_Measure_Anything__Finding_the_Value_of_Intangibles_in_Business__Second_Edition.pdf--><i>How to Measure Anything Finding the Value of Intangibles in Business</i></a>. 2ed. Wiley, 2010.</div>

<div class="csl-entry" id="Jeffery2002"><NAME>. <a href="http://www.princeton.edu/~bayesway/Book*.pdf"><i>Subjective Probability: The Real Thing</i></a>. Cambridge University Press, 2002.</div>
  
<div class="csl-entry" id="Kahneman2011"><NAME>. <a href="http://www.nytimes.com/2011/10/23/magazine/dont-blink-the-hazards-of-confidence.html">"Dont Blink! The Hazards of Confidence."</a> <i>The New York Times</i>, October 19, 2011, sec. Magazine.</div>
  
<div class="csl-entry" id="KassinFong1999">Kassin, <NAME>., and <NAME>. <a href="http://web.williams.edu/Psychology/Faculty/Kassin/files/kassin_fong_1999.pdf">"Im Innocent!: Effects of Training on Judgments of Truth and Deception in the Interrogation Room."</a> <i>Law and Human Behavior</i> 23, no. 5 (October 1, 1999): 499516. doi:10.1023/A:1022330011811.</div>

<div class="csl-entry" id="Knight1921">Knight, <NAME>. (<NAME>). <a href="http://www.amazon.com/gp/product/1602060053/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=1602060053&linkCode=as2&tag=sethrylan-20"><!--http://www.econlib.org/library/Knight/knRUP.html--><i>Risk, Uncertainty and Profit</i></a>. Boston, New York, Houghton Mifflin Company, 1921.</div>

<div class="csl-entry" id="LichtensteinFischhoff1978">Lichtenstein, Sarah, and <NAME>. <a href="http://www.dtic.mil/dtic/tr/fulltext/u2/a069703.pdf"><i>Training for Calibration</i></a>, November 1978.</div>
  
<div class="csl-entry" id="LichtensteinEtAl1982">Lichtenstein, Sarah, <NAME>, and <NAME>. <a href="http://www.dtic.mil/dtic/tr/fulltext/u2/a101986.pdf">"Calibration of Probabilities: The State of the Art to 1980."</a> In <a href="http://www.amazon.com/gp/product/0521284147/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0521284147&linkCode=as2&tag=sethrylan-20"><i>Judgment Under Uncertainty: Heuristics and Biases</i></a>, edited by <NAME>, <NAME>, and <NAME>, 306334. Cambridge, UK: Cambridge University Press, 1982.</div>

<div class="csl-entry" id="LindeyEtAl1979"><NAME>., <NAME>, and <NAME>. <a href="citations/Lindly_et_al-On_the_Reconciliation_of_Probability_Assessments.pdf">"On the Reconciliation of Probability Assessments."</a> <i>Journal of the Royal Statistical Society. Series A (General)</i> 142, no. 2 (January 1, 1979): 146180. doi:10.2307/2345078.</div>

<div class="csl-entry" id="Marx2013"><NAME>.  <a href="http://www.nature.com/nmeth/journal/v10/n7/full/nmeth.2530.html">"Data Visualization: Ambiguity as a Fellow Traveler."</a> <i>Nature Methods</i> 10, no. 7 (July 2013): 613615. doi:10.1038/nmeth.2530.</div>
  
<div class="csl-entry" id="McIntyre2007"><NAME>. <a href="http://www.atm.damtp.cam.ac.uk/mcintyre/mcintyre-thinking-probabilistically.pdf">"On Thinking Probabilistically."</a> In <i>Extreme Events (Proc. 15th Aha Hulikoa Workshop)</i>, 153161. U. of Hawaii: SOEST, 2007.</div>
 
<div class="csl-entry" id="Oskamp1965"><NAME>. <a href="citations/Oskamp-Overconfidence_in_Case_Study_Judgements.pdf">"Overconfidence in Case-study Judgments."</a> <i>Journal of Consulting Psychology</i> 29, no. 3 (1965): 261265. doi:10.1037/h0022125.</div>

<div class="csl-entry" id="Plous1993"><NAME>. <a href="http://www.amazon.com/gp/product/0070504776/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0070504776&linkCode=as2&tag=sethrylan-20"><i>The Psychology of Judgment and Decision Making</i></a>. New York: McGraw-Hill, 1993.</div>

<div class="csl-entry" id="RadzevickMoore2009"><NAME>., and <NAME>. <a href="http://www.gsb.stanford.edu/sites/default/files/documents/ob_01_09_moore.pdf">"Competing to Be Certain (but Wrong): Social Pressure and Overprecision in Judgment."</a> <i>Academy of Management Proceedings</i> 2009, no. 1 (August 1, 2009): 16. doi:10.5465/AMBPP.2009.44246308.</div>

<div class="csl-entry" id="Silver2012"><NAME>. <a href="http://www.amazon.com/gp/product/159420411X/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=159420411X&linkCode=as2&tag=sethrylan-20"><i>The Signal and the Noise: Why So Many Predictions Fail  but Some Dont</i></a>. 1ed. Penguin Press HC, The, 2012.</div>

<div class="csl-entry" id="Wilson1994"><NAME>. <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.71.4909&rep=rep1&type=pdf">"Cognitive Factors Affecting Subjective Probability Assessment,"</a> 1994.</div>

                        </td>
                        <td width="20%" valign="top">
                        </td>
                    </tr>
                </table>
            </div>

        </div>
        <script src="//ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
        <script src="//ajax.googleapis.com/ajax/libs/jqueryui/1.10.3/jquery-ui.min.js"></script>        
        <script src="js/jquery.ui.touch-punch.min.js"></script>
        <script src="js/jquery.qtip.js"></script>
{'fraction_non_alphanumeric': 0.12904265359206316, 'fraction_numerical': 0.04990447252721957, 'mean_word_length': 2.9205847847667177, 'pattern_counts': {'":': 0, '<': 861, '<?xml version=': 0, '>': 861, 'https://': 2, 'lorem ipsum': 0, 'www.': 44, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1600753', 'n_tokens_mistral': 14477, 'n_tokens_neox': 12152, 'n_words': 3501}