<jupyter_start><jupyter_text>Bite Size BayesCopyright 2020 <NAME>License: [Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)](https://creativecommons.org/licenses/by-nc-sa/4.0/)  Review[In the previous notebook](https://colab.research.google.com/github/AllenDowney/BiteSizeBayes/blob/master/04_dice.ipynb) I presented Theorem 4, which is a way to compute the probability of a disjunction (`or` operation) using the probability of a conjunction (`and` operation).$P(A ~or~ B) = P(A) + P(B) - P(A ~and~ B)$Then we used it to show that the sum of the unnormalized posteriors is the total probability of the data, which is why the Bayes table works.We saw several examples involving dice, and I used them to show how prediction and inference are related: the Bayes table actually solves the prediction problem on the way to solving the inference problem.  BayesvilleIn this notebook we'll consider a famous example where Bayes's Theorem brings clarity to a confusing topic: medical testing.<NAME> explains the scenario in this video from Stat110x at HarvardX.(You might have to run the following cell to see the video.)<jupyter_code>from IPython.display import YouTubeVideo
YouTubeVideo('otdaJPVQIgg')<jupyter_output><empty_output><jupyter_text>I'll paraphase the problem posed in the video: > In Bayesville, 1% of the population has an undiagnosed medical condition.  Jimmy gets tested for the condition and the test comes back positive; that is, the test says Jimmy has the condition.  >> The test is 95% accurate, which means>> * If you give the test to someone with the condition, the probability is 95% that the test will be positive, and >> * If you give it to someone who does not have the condition, the probability is 95% that the test will be negative.>> What is the probability that Jimmy actually has the condition?  Because the test is 95% accurate, it is tempting to say that the probability is 95% that the test is correct and Jimmy has the condition. But that is wrong.  Or maybe I should say it's the right answer to a different question.  95% is the probability of a positive test, given a patient with the condition.  But that's not what the question asked, or what Jimmy wants to know.To Jimmy, the important question is the probability he has the condition, given a positive test.  As we have seen, and as Joe explains in the video:$P(A|B) ≠ P(B|A)$We can use a Bayes table to answer Jimmy's question correctly.  Bayes tableI'll use two strings to represent the hypotheses: `condition` and `no condition`.The prior for `condition` is the probability a random citizen of Bayesville has the condition, which is 1%.The prior for `no condition` is the probability that a random citizen does not have the disease, which is 99%.Let's put those values into a Bayes table.<jupyter_code>import pandas as pd

table = pd.DataFrame(index=['condition', 'no condition'])
table['prior'] = 0.01, 0.99
table<jupyter_output><empty_output><jupyter_text>The data is the positive test, so the likelihoods are:* The probability of a correct positive test, given the condition, which is 95%.* The probability of an incorrect positive test, given no condition, which is 5%.<jupyter_code>table['likelihood'] = 0.95, 0.05
table<jupyter_output><empty_output><jupyter_text>Once we have priors and likelihoods, the remaining steps are always the same.  We compute the unnormalized posteriors:<jupyter_code>table['unnorm'] = table['prior'] * table['likelihood']
table<jupyter_output><empty_output><jupyter_text>And the total probability of the data.<jupyter_code>prob_data = table['unnorm'].sum()
prob_data<jupyter_output><empty_output><jupyter_text>Then divide through to get the normalized posteriors.<jupyter_code>table['posterior'] = table['unnorm'] / prob_data
table<jupyter_output><empty_output><jupyter_text>The posterior for `condition` is substantially higher than the prior, so the positive test is evidence in favor of `condition`.But the prior is small and the evidence is not strong enough to overcome it; despite the positive test, the probability that Jimmy has the condition is only about 16%.Many people find this result surprising and some insist that the probability is 95% that Jimmy has the condition.The mistake they are making is called the [base rate fallacy](https://en.wikipedia.org/wiki/Base_rate_fallacy) because it ignores the "base rate" of the condition, which is the prior.  Put a function on itAt this point you might be sick of seeing the same six lines of code over and over, so let's put them in a function where you will never have to see them again.<jupyter_code>def make_bayes_table(hypos, prior, likelihood):
    """Make a Bayes table.
    
    hypos: sequence of hypotheses
    prior: prior probabilities
    likelihood: sequence of likelihoods
    
    returns: DataFrame
    """
    table = pd.DataFrame(index=hypos)
    table['prior'] = prior
    table['likelihood'] = likelihood
    table['unnorm'] = table['prior'] * table['likelihood']
    prob_data = table['unnorm'].sum()
    table['posterior'] = table['unnorm'] / prob_data
    return table<jupyter_output><empty_output><jupyter_text>This function takes three parameters:* `hypos`, which should be a sequence of hypotheses.  You can use almost any type to represent the hypotheses, including string, `int`, and `float`.* `prior`, which is sequence of prior probabilities, $P(H)$  for each $H$.* `likelihood`, which is a sequence of likelihoods, $P(D|H)$  for each $H$.All three sequences should be the same length.Here's a solution to the previous problem using `make_bayes_table`:<jupyter_code>hypos = ['condition', 'no condition']
prior = 0.01, 0.99
likelihood = 0.95, 0.05

make_bayes_table(hypos, prior, likelihood)<jupyter_output><empty_output><jupyter_text>**Exercise:** Suppose we take the same test to another town, called Sickville, where the base rate of the disease is 10%, substantially higher than in Bayesville.  If a citizen of Sickville tests positive, what is the probability that they have the condition?Use `make_bayes_table` to compute the result.<jupyter_code># Solution goes here<jupyter_output><empty_output><jupyter_text>With a higher base rate, the posterior probability is substantially higher. **Exercise:** Suppose we go back to Bayesville, where the base rate is 1%, with a new test that is 99.5% accurate.If a citizen of Bayesville tests positive with the new test, what is the probability they have the condition?Use `make_bayes_table` to compute the result.<jupyter_code># Solution goes here<jupyter_output><empty_output><jupyter_text>With an accuracy of 99.5%, the positive test provides stronger evidence, so it is able to overcome the small prior.  The Elvis problemHere's a problem from [*Bayesian Data Analysis*](http://www.stat.columbia.edu/~gelman/book/):> <NAME> had a twin brother (who died at birth).  What is the probability that Elvis was an identical twin?For background information, I used data from theU.S. Census Bureau, [Birth, Stillbirth, and Infant Mortality Statistics for the Continental United States, the Territory of Hawaii, the Virgin Islands 1935](https://www.cdc.gov/nchs/data/vsushistorical/birthstat_1935.pdf) to estimate that in 1935, about 1/3 of twins were identical.**Exercise:** Use this base rate and a Bayes table to compute the probability that Elvis was an identical twin.Hint: Because identical twins have the same genes, they are almost always the same sex.<jupyter_code># Solution goes here<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07863521258163401, 'fraction_numerical': 0.011462081833933093, 'mean_word_length': 5.284757118927973, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 58, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '24936', 'license': 'MIT', 'n_tokens_mistral': 2149, 'n_tokens_neox': 2017, 'n_words': 966, 'path': '05_test.ipynb', 'repo_name': 'jandolezal/BiteSizeBayes', 'size': '7527'}