<jupyter_start><jupyter_text>Confidence Intervals and BootstrappingTo use this tutorial, read the commands and execute the code in each cell.The learning objective is to understand how to compute confidence intervals (known as credible intervals when using Bayesian approaches), which are a fundamental piece of rigorous, quantitative science beacuse they describe the uncertainty that you have in estimating some property of the world given noisy, finite observations.For NGG students, this tutorial is meant to be used in tandem with this Discussion on the NGG Canvas site:https://canvas.upenn.edu/courses/1358934/discussion_topics/5464266Copyright 2019 by <NAME>, University of Pennsylvania        Created 09/21/18        Updated 12/31/19        Ported to Python 04/23/21<jupyter_code>import platform

# Output on system used for development/testing:
# 3.9.2
print(platform.python_version())

# Uncomment and run to clear workspace
# %reset
import scipy.stats as st
import numpy as np

# Exercise: Compute confidence/credible intervals for simulated data sampled from a population that is Gaussian distributed with mean mu=10 and standard deviation sigma=2, for n=5, 10, 20, 40, 80 at a 95% confidence level.
mu = 10
sigma = 2
alpha = 0.95
num_bootstraps = 1000

# Loop through the n's. Note that the different approaches converge on the same answer as n gets large
for n in [5, 10, 20, 40, 80, 160, 1000]:
   
   # Simulate some data
   samples = np.random.normal(mu, sigma, n)
   
   # Save the mean
   sample_mean = np.mean(samples)
   
   # Show the mean, n
   print(f'n = {n}, mean = {sample_mean:.2f}')
   
   # Method 1: analytic solution assuming Gaussian
   #
   # Get the z-score for the given confidence level (make it negative so we can subtract it to make the lower interval)
   z = -st.norm.ppf((1-alpha)/2)

   # 1a. Use the given sigma
   sem = sigma/np.sqrt(n)
   print(f'1a: CI=[{sample_mean-sem*z:.2f}, {sample_mean+sem*z:.2f}]')
      
   # 1b. Use the sample sigma
   #     BEST IF n IS LARGE (>30)
   sem = np.std(samples)/np.sqrt(n)
   print(f'1b: CI=[{sample_mean-sem*z:.2f}, {sample_mean+sem*z:.2f}]')
   
   # Method 2: analytic solution assuming t-distribution
   #      BEST IF n IS SMALL (<30) ... note that as n increases, the t distribution approaches a Gaussian and methods 1 and 2 become more and more similar

   # Get the cutoff using the t distribution, which is said to have n-1 degrees of freedom
   t = -st.t.ppf((1-alpha)/2,df=n-1)
   sem = np.std(samples)/np.sqrt(n);
   print(f'2 : CI=[{sample_mean-sem*t:.2f}, {sample_mean+sem*t:.2f}]')
   
   # Method 3: bootstrap!
   # Resample the data with replacement to get new estimates of mu 
   # Note that here we do not make any assumptions about the nature of the real distribution.
   mu_star = [np.mean(np.random.choice(samples, size=n)) for ii in np.arange(num_bootstraps)]
   
   # Now report the CI directly from the bootstrapped distribution
   print(f'3 : CI=[{np.percentile(mu_star, 100*(1-alpha)/2):.2f}, {np.percentile(mu_star, 100*(alpha+(1-alpha)/2)):.2f}]')
            
   # Method 4: Credible interval
   # See the Canvas discussion -- under these assumptions (i.e., data generated from a Gaussian distribution with known sigma), the answer is exactly the same as with Method 1, above. Note that this equivalence is NOT true in general, which means that frequentist confidence intervals and Bayesian credible intervals can givedifferent answers for certain distributions.
   
   # Formatting
   print(f'----')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09362662154540327, 'fraction_numerical': 0.034122955442752394, 'mean_word_length': 3.7356475300400533, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '247536', 'license': 'Apache-2.0', 'n_tokens_mistral': 1172, 'n_tokens_neox': 1043, 'n_words': 430, 'path': 'old/Confidence Intervals/ConfidenceIntervals.ipynb', 'repo_name': 'PennNGG/Statistics', 'size': '3554'}