<jupyter_start><jupyter_text>Frequentist versus Bayesian StatisticsTo use this tutorial, read the commands and execute the code cell-by-cell.The learning objective is to gain insights into thinking about inference from a "Frequentist" versus a "Bayesian" perspective. In brief, because a Frequentist does not consider the probability of an event or state of the world or hypothesis, only their frequency of occurrance, it is not possible to ask questions of the form "what is the probabilty that hypothesis x is true?" Instead, one can only consider questions of the form, "what is the probabilty that I would have obtained my data, given that hypothesis x is true?" In contrast, Bayesians consider the probabilities of such things (often called the strength of belief), but doing so can require making assumptions that can be difficult to prove.For NGG students, this tutorial is meant to be used in tandem with this Discussion on the NGG Canvas site:https://canvas.upenn.edu/courses/1358934/discussion_topics/5440322Copyright 2019 by <NAME>, University of Pennsylvania<jupyter_code>import platform

# Output on system used for development/testing:
# 3.9.2
print(platform.python_version())

# Uncomment and run to clear workspace
# %reset<jupyter_output><empty_output><jupyter_text>Let's start with a simple example, taken from:https://en.wikipedia.org/wiki/Base_rate_fallacyExample_1:_HIV"Imagine running an HIV test on" A SAMPLE "of 1000 persons ...""The test has a false positive rate of 5% (0.05)..." i.e., the probability that someone who takes the test gets a POSITIVE result despite the fact that the person does NOT have HIV"...and no false negative rate." i.e., The probability that someone who takes the test gets a NEGATIVE result despite the fact that the person DOES have HIV  Question 1: If someone gets a positive test, is it "statistically significant"?**Answer**: Statistical significance from the Frequentist perspective is typically measured by comparing p to a threshold value; e.g., p<0.05. In this case, "p" is shorthand for "the probabilty of obtaining the data under the Null Hypothesis", so we are checking for: $$p(data\,|\,Null\,Hypothesis) < 0.05$$Here we take the Null Hypothesis as "not infected", and the data are just the single positive test. Therefore, the relvant p-value is simply the false-positive rate: p=0.05, which is typically considered "not significant." However, you can also see that it is not particularly informative.<jupyter_code>N = 10000                   # size of the SAMPLE
false_positive_rate = 0.05 
false_negative_rate = 0

print(f'The probability of obtaining the data under the Null Hypothesis = {false_positive_rate}')<jupyter_output>The probability of obtaining the data under the Null Hypothesis = 0.05<jupyter_text>Question 2: What is the probability that if someone gets a positive test, that person is infected?**Answer**: Here we are asking for a different probability: $$p(infected\,|\,positive\,test) = p(hypothesis\,|\,data)$$ which is the "posterior probability" of the hypothesis, given the data.Let's work our way backwards to figure out what information we need to solve this problem. We can compute the probability that someone with a positive test is infected from a particular population as:$$probability\,infected\,given\,positive\,test = \frac{total(infected\,and\,postive)}{total(positive)}$$It should be obvious that to compute this quantity, we need to know the number of people in the population who are actually infected, in addition to knowing the number of people who had a positive test.<jupyter_code>import scipy.stats as st
import numpy as np

# So let's start by definining how many in the population are actually infected. We'll start by assuming that that *real* rate of infection is 0.5 (i.e., half the POPULATION is infected), and then do a quick simulation to find out how many in our SAMPLE of N people are infected. We can do this simulation by by getting N picks from a binomial distribution, where each pick determines "isInfected" for a single person according to the assumed rate of infection:
is_infected = st.binom.rvs(1, 0.5, size=N)

# Now we can count the number infected
num_infected = is_infected.sum()

# Now we need to count the number of people who got a positive test in this example. There is no false negative rate, which implies that everyone who is infected got a positive test:
is_positive = np.copy(is_infected)

# But there is a non-zero false-positive rate, which implies that some of the people who are **not** infected will also have a positive test. We can use binornd again to generate random picks from a binomial distribtuion according to the false-positive rate:
is_positive[is_infected==0] = st.binom.rvs(1, false_positive_rate, size=N-num_infected)

# Now we can compute the probability that someone with a positive test is infected:
p_is_infected_given_is_positive = (np.logical_and(is_infected==1, is_positive==1).sum())/is_positive.sum()
print(f'Probaility infected given a positive test = {p_is_infected_given_is_positive:.4f}')<jupyter_output>Probaility infected given a positive test = 0.9527<jupyter_text>Let's do the same thing, but this time we will try different values for the proportion of the population that is actually infected. What you should notice is that the **PROPORTION INFECTED GIVEN A POSITIVE TEST** depends (a lot!) on the **OVERALL RATE OF INFECTION**. Put another way, to determine the probabilty of a hypothesis, given your data (e.g., proportion infected given a positive test), you have to know the probability that the hypothesis was true without any data.Why is this the case? It is a simple consequence of the definition of a conditional probability, formulated as Bayes' Rule. Specifically, the joint probability of two events, call them A and B, is defined as: $$p(A\,and\,B) = p(A) \times p(B\,|\,A)$$ $$p(B\,and\,A) = p(B) \times p(A\,|\,B)$$Now, calling A the Hypothesis and B the Data, then rearranging, we get:$$p(Hypothesis\,|\,Data) = \frac{p(Data\,|\,Hypothesis) \times p(Hypothesis)}{p(Data)}$$So you cannot calculate the probability of the hypothesis, given the data (i.e., the Bayesian posterior), without knowing the probability of the hypothesis independent of any data (i.e., the prior)<jupyter_code>infected_proportions = np.arange(0.0, 1.1, 0.1)
for idx, val in enumerate(infected_proportions):
    
   # Simulate # infections in the SAMPLE, given the POPULATION rate
   is_infected = st.binom.rvs(1, val, size=N)
      
   # Count the number infected
   num_infected = is_infected.sum()
   
   # Make array of positive tests, given that falseNegativeRate=0 ...
   is_positive = np.copy(is_infected)
   
   # And falsePositiveRate > 0
   is_positive[is_infected==0] = st.binom.rvs(1, false_positive_rate, size=N-num_infected)
   
   # The probability that someone with a positive test is infected
   p_is_infected_given_is_positive = (np.logical_and(is_infected==1, is_positive==1).sum())/is_positive.sum()
   
   # We can compute the Bayesian Posterior as:
   # p(hypothesis | data) = (p(data | hypothesis) * p(hypothesis)) / p(data)
   # Note that we are using the true rate from the full POPULATION, so these predictions will differ slightly from the probability computed above (pIsInfectedGivenIsPositiveTest) from the SAMPLE
   p_data_given_hypothesis = 1 - false_negative_rate
   p_hypothesis = val
   p_data = is_positive.sum()/is_positive.size   
   p_hypothesis_given_data = (p_data_given_hypothesis * p_hypothesis) / p_data

   # Compute the theoretial posterior probability: 
   print(f'Infection rate={val:.1f}, proportion infected given a positive test={p_is_infected_given_is_positive:.3f}, Posterior={p_hypothesis_given_data:.3f}')<jupyter_output>Infection rate=0.0, proportion infected given a positive test=0.000, Posterior=0.000
Infection rate=0.1, proportion infected given a positive test=0.699, Posterior=0.672
Infection rate=0.2, proportion infected given a positive test=0.821, Posterior=0.840
Infection rate=0.3, proportion infected given a positive test=0.895, Posterior=0.888
Infection rate=0.4, proportion infected given a positive test=0.928, Posterior=0.933
Infection rate=0.5, proportion infected given a positive test=0.956, Posterior=0.944
Infection rate=0.6, proportion infected given a positive test=0.970, Posterior=0.952
Infection rate=0.7, proportion infected given a positive test=0.980, Posterior=0.970
Infection rate=0.8, proportion infected given a positive test=0.988, Posterior=0.988
Infection rate=0.9, proportion infected given a positive test=0.992, Posterior=0.991
Infection rate=1.0, proportion infected given a positive test=1.000, Posterior=1.000
{'fraction_non_alphanumeric': 0.07963581883139334, 'fraction_numerical': 0.022473205024778148, 'mean_word_length': 5.047386759581881, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 16, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '575497', 'license': 'Apache-2.0', 'n_tokens_mistral': 2667, 'n_tokens_neox': 2389, 'n_words': 1080, 'path': 'old/Frequentist Vs Bayesian/FrequentistVsBayesian.ipynb', 'repo_name': 'PennNGG/Statistics', 'size': '8685'}