<jupyter_start><jupyter_text>*Data Science Unit 1 Sprint 3 Lesson 1* Statistics, Probability and InferenceEver thought about how long it takes to make a pancake? Have you ever compared the tooking time of a pancake on each eye of your stove? Is the cooking time different between the different eyes? Now, we can run an experiment and collect a sample of 1,000 pancakes on one eye and another 800 pancakes on the other eye. Assumed we used the same pan, batter, and technique on both eyes. Our average cooking times were 180 (5 std) and 178.5 (4.25 std) seconds repsectively. Now, we can tell those numbers are not identicial, but how confident are we that those numbers are practically the same? How do we know the slight difference isn't caused by some external randomness?Yes, today's lesson will help you figure out how long to cook your pancakes (*theoretically*). Experimentation is up to you; otherwise, you have to accept my data as true. How are going to accomplish this? With probability, statistics, inference and maple syrup (optional).  Learning Objectives* [Part 1](p1): Normal Distribution Revisted* [Part 2](p2): Student's T Test* [Part 3](p3): Hypothesis Test & Doing it Live  Normal Distribution RevisitedWhat is the Normal distribution: A probability distribution of a continuous real valued random-variable. The Normal distribution properties make it useful for the *Central Limit Theorm*, because if we assume a variable follows the normal distribution, we can make certain conclusions based on probabilities.<jupyter_code>import numpy as np
#Normal distribution is described as two terms: 1. mean and 2.std
N = 1000
mu = 0 # mean
sigma = 0.1 # standard deviation

sample = np.random.normal(mu, sigma, N) #tool for generating or processing old tools

# Verify the mean of our sample
abs(mu - np.mean(sample)) < 0.01
sample[:10]
mu - np.mean(sample) #this number should be very small
#the means should be virtually the same
#Variance
sigma - np.std(sample, ddof=1)
# Verify the variance/std of our sample
abs(sigma - np.std(sample, ddof=1)) < 0.01
#manually doing the std dev
var = (1/(N-1))*np.sum((sample - np.mean(sample))**2) #this is our variance
#Take the square root of the variance to yield the std dev
np.sqrt(var)
np.std(sample)
import seaborn as sns
from matplotlib import style #so that we can use a different style graphic

style.use('fivethirtyeight')
ax = sns.distplot(sample, color='purple')
ax.axvline(np.percentile(sample, 97.5),0) #adds the vertical lines to show where the third std deviation is
ax.axvline(np.percentile(sample, 2.5), 0)
#quat-quat plot to determine if it is normally distributed
#first, generate a normal data set. Not necessary as most libraries do this for us.
#but this will show us how to do this manually

#generate random but normally distributed data first
sample_norm = np.random.normal(size=1000) #defaults to mean of 0 and std of 1
#sns.distplot(sample_norm)
sample_norm
#to run the qq plot, sort the values, from lowest to highest
a = sorted(sample_norm)
b = sorted(sample)
print(a)
b
sns.scatterplot(a,b) #pretty straight line...suggesting normal dist.<jupyter_output>[-2.5545138573989683, -2.3583458882527424, -2.3411370264670244, -2.338061382476589, -2.270719954829626, -2.2542879844038732, -2.2413872198235545, -2.1698628285296224, -2.148348177658662, -2.14371207084644, -2.1233801683340157, -2.11468394848517, -2.0839436778728695, -2.048640208399139, -2.047641554278812, -2.031752999504663, -2.029437804750803, -2.0204118992356386, -2.0144664351964936, -2.0051559916750517, -1.9905190462890872, -1.984808092512009, -1.9709863124736915, -1.967835079588467, -1.9655812830909127, -1.9363515122331159, -1.9054623875066818, -1.8816034518684492, -1.8015090298341256, -1.7904561128743508, -1.787923518253836, -1.781623706828108, -1.7791234681496637, -1.7763872175407425, -1.7748854361731579, -1.76927411581499, -1.7523834098017201, -1.7503500809150332, -1.7390367682051873, -1.712358830852667, -1.7122018015343436, -1.6877004652604204, -1.6725662552258065, -1.6678528994231092, -1.662702714251437, -1.6406523057628077, -1.6297341594798074, -1.627927749416486, -1.62200696[...]<jupyter_text>Student's T Test>Assuming data come from a Normal distribution, the t test provides a way to test whether the sample mean (that is the mean calculated from the data) is a good estimate of the population mean. The derivation of the t-distribution was first published in 1908 by <NAME> while working for the Guinness Brewery in Dublin. Due to proprietary issues, he had to publish under a pseudonym, and so he used the name Student.The t-distribution is essentially a distribution of means of normaly distributed data. When we use a t-statistic, we are  checking that a mean fails within a certain $\alpha$ probability of the mean of means.<jupyter_code>t_df10 = np.random.standard_t(df=10, size=10)
t_df100 = np.random.standard_t(df=100, size=100)
t_df1000 = np.random.standard_t(df=1000, size=1000)
sns.kdeplot(t_df10, color='r');
sns.kdeplot(t_df100, color='y');
sns.kdeplot(t_df1000, color='b');
i = 10
for sample in [t_df10, t_df100, t_df1000]:
    print(f"t - distribution with {i} degrees of freedom")
    print("---" * 10)
    print(f"Mean: {sample.mean()}")
    print(f"Standard Deviation: {sample.std()}")
    print(f"Variance: {sample.var()}")
    i = i*10<jupyter_output>t - distribution with 10 degrees of freedom
------------------------------
Mean: -0.5662711126999362
Standard Deviation: 1.1275052143375808
Variance: 1.271268008358434
t - distribution with 100 degrees of freedom
------------------------------
Mean: -0.0795330190236767
Standard Deviation: 1.06049329175301
Variance: 1.1246460218531351
t - distribution with 1000 degrees of freedom
------------------------------
Mean: -0.04246118427001148
Standard Deviation: 0.9818815900385561
Variance: 0.9640914568566432<jupyter_text>Why is it different from normal? To better reflect the tendencies of small data and situations with unknown population standard deviation. In other words, the normal distribution is still the nice pure ideal (thanks to the central limit theorem), but the t-distribution is much more useful in many real-world situations.  Live Lecture - let's perform and interpret a t-testWe'll generate our own data, so we can know and alter the "ground truth" that the t-test should find. We will learn about p-values and how to interpret "statistical significance" based on the output of a hypothesis test. We will also dig a bit deeper into how the test statistic is calculated based on the sample error, and visually what it looks like to have 1 or 2 "tailed" t-tests.<jupyter_code># TODO - during class, but please help!

from scipy.stats import ttest_ind, ttest_ind_from_stats, ttest_rel

groupA = np.random.normal(180, 5, 1000)
groupB = np.random.normal(178.5, 4.25, 800)

groupA[:10]
groupB[:10]
#get the basic information for each of the samples
for sample in [groupA, groupB]:
    print(f'Mean: {sample.mean()}')
    print(f'Standard Deviation: {sample.std()}')
    print(f'Variance: {sample.var()}')
    print('----' * 10)
#now we will run the ttst for ind samples
statistic, pvalue = ttest_ind(groupA, groupB)
print(statistic, pvalue)
#the greater the 'statistic' the more difference that exists between the two samples
#pvalue is between 0 and 1
#small pvalue suggests that there is not enough evidence to reject the null hypothesis
#null hypothesis is some prior knowledge or belief about the distributions
pvalue < 0.05
#if pvalue is big ---> you cannot reject null hypothesis
#if pvalue is small ---> you can reject null hypothesis, in favor of an alternative hypothesis<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0993602297950124, 'fraction_numerical': 0.14531923227575402, 'mean_word_length': 4.6448047162859245, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '195260', 'license': 'MIT', 'n_tokens_mistral': 3088, 'n_tokens_neox': 2416, 'n_words': 946, 'path': 'module1-statistics-probability-and-inference/LS_DS_131_Statistics_Probability_and_Inference.ipynb', 'repo_name': 'echiyembekeza/DS-Unit-1-Sprint-3-Statistical-Tests-and-Experiments', 'size': '7667'}