<jupyter_start><jupyter_text>IntroductionInferential Statistics is the process of examining the observed data (sample) in order to make conclusions about properties/parameter of a Population. The conclusion of a statistical inference is called a *statistical proposition*. Some common forms are the following:+ a point estimate (mean)+ an interval estimate (confidence interval)+ rejection of a hypothesis (hypothesis testing)+ clustering or classification of individual data points into groups (Machine Learning techniques, Regression, Classification)The classic approach for Inferential Statistics requires to make assumptions about the population distribution - which is **unknown**. If we knew about the Population, there would be no need for Inference!Hence, using analytical formulas - and making wild assumptions while at it - is not advised. Especially nowadays: with the available processing power and so many readily available programming tools, we can easily make inferences over simulated data distribution.This notebook will cover **Confidence Interval**.  Confidence IntervalImporting necessary libraries and loading the example dataset. This fictitious dataset contains the average height (in centimeters) of undergrad students, as well as categorical information about age and if they drink coffee.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
plt.rcParams['axes.titleweight']= 'bold'

np.random.seed(42)

df_full = pd.read_csv('undergrad_students.csv')
df_full.head(3)<jupyter_output><empty_output><jupyter_text>We are interested in studying whether the average height for coffee drinkers is the same as for non-coffee drinkers. $$ Height_{Coffee} = Height_{No \space Coffee} $$ Our research question can be rewritten as:The difference between the average height of Coffee drinkers and No Coffee drinker is equal to zero. $$ Height_{Coffee} - Height_{No \space Coffee} = 0$$ First, let's create a sample of size 200 from our full dataset. We will call it the "*Original Sample*".Then, we can use the Bootstrapping technique to generate the Sampling Distribution based on the Original Sample.Finally, we calculate the average height of each group of interest (coffee drinker vs no coffee), compute their difference, and store it in a list called `diffs`. We repeat this process 10,000 times so the Central Limit Theory "kicks-in", resulting in a normally distributed difference in average heights.> **Note**: this process yields 3 arrays of size 10,000: average height of coffee drinkers; average height of those who do not drink coffee, and the difference in means.<jupyter_code>df_sample = df_full.sample(200)
# instantiate empty list
diffs = []

# Bootstraping 10,000 times
for _ in range(10_000):
    boot_sample = df_sample.sample(len(df_sample), replace=True)
    avg_height_coffee = boot_sample.query('drinks_coffee == True').height.mean()
    avg_height_not_coffee = boot_sample.query('drinks_coffee == False').height.mean()
    diffs.append(avg_height_coffee - avg_height_not_coffee)
print(f'Sampling Distribution Mean: {np.mean(diffs):.3f} cm')
print(f'Sampling Distribution Std: {np.std(diffs):.3f} cm')
# Actual difference in average height, based on the original sample
height_coffee = df_sample.query('drinks_coffee == True').height
height_no_coffee = df_sample.query('drinks_coffee == False').height

actual_avg_height_difference = height_coffee.mean() - height_no_coffee.mean()
print(f'Original Sample average height difference: {actual_avg_height_difference:.3f} cm')<jupyter_output>Original Sample average height difference: 3.390 cm<jupyter_text>Checking the difference between the average height of both groups, we can see the mean of our Sampling Distribution (3.394 cm) closely approximates the actual  average height difference observed in the original sample (3.390 cm).  Calculating Lower and Upper BoundsLastly, let's find the lower and upper bounds of the interval containing the correct parameter with a confidence level of 95%: $$\alpha = 0.05$$<jupyter_code>alpha = 5.0
lower, upper = np.percentile(diffs, alpha/2) , np.percentile(diffs, 100 - alpha/2)
print(f'Confidence Interval: [{lower:.2f}, {upper:.2f}]')
# Plotting
ax = sns.distplot(diffs, hist_kws={'edgecolor':'gray'}, label='Sampling Distribution')

line = ax.lines[0]
line = line.get_xydata()
interval = line[(line[:,0] > lower) & (line[:,0] < upper)]
x1 = interval[:,0]
y1 = interval[:,1]

ax.fill_between(x1, y1, color='r', alpha=0.4, hatch='//',\
                label='Confidence Interval')

plt.title('Normal Distribution: Difference in Average Height')
plt.xlabel('height difference [cm]')
plt.ylabel('Density');

plt.axvline(lower, color='r', ls=':', label='Extreme Bounds')
plt.axvline(upper, color='r', ls=':')
plt.legend();
CI_width = upper - lower
print(f'Average Height Difference: {np.mean(diffs):.2f} cm')
print(f'Margin of error: {CI_width/2:.2f} cm')<jupyter_output>Average Height Difference: 3.39 cm
Margin of error: 2.34 cm
{'fraction_non_alphanumeric': 0.09387430389817025, 'fraction_numerical': 0.015910898965791568, 'mean_word_length': 5.832880434782608, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '245139', 'license': 'MIT', 'n_tokens_mistral': 1496, 'n_tokens_neox': 1378, 'n_words': 557, 'path': '1_ Inference - Confidence Interval/Statistical Inference - Confidence Interval.ipynb', 'repo_name': 'marcellovictorino/Practical-Statistics', 'size': '5028'}