<jupyter_start><jupyter_text>Intro to Statistics This class, *Intro to Statistics*, builds on probability theory to enable us to quantify our confidence about how distributions of data are related to one another. Through the measured exposition of theory paired with interactive examples, you’ll develop a working understanding of all of the essential statistical tests for assessing whether data are correlated with each other or sampled from different populations -- tests which frequently come in handy for critically evaluating the inputs and outputs of machine learning algorithms. You’ll also learn how to use regression to make predictions about the future based on training data. The content covered in this class builds on the content of other classes in the *Machine Learning Foundations* series (linear algebra, calculus, and probability theory) and is itself foundational for the *Optimization* class.  Over the course of studying this topic, you'll: * Develop an understanding of what’s going on beneath the hood of predictive statistical models and machine learning algorithms, including those used for deep learning. * Hypothesize about and critically evaluate the inputs and outputs of machine learning algorithms using essential statistical tools such as the t-test, ANOVA, and R-squared. * Use historical data to predict the future using regression models that take advantage of frequentist statistical theory (for smaller data sets) and modern machine learning theory (for larger data sets), including why we may want to consider applying deep learning to a given problem. **Note that this Jupyter notebook is not intended to stand alone. It is the companion code to a lecture or to videos from Jon Krohn's [Machine Learning Foundations](https://github.com/jonkrohn/ML-foundations) series, which offer detail on the following:***Segment 1: Review of Probability Theory** Frequentist vs Bayesian Statistics* Normal Distributions* Measures of Central Tendency * Quantiles* Variance, Standard Deviation, and Standard Error* Covariance and Correlation* The Central Limit Theorem*Segment 2: Quantifying Confidence** z-scores* The t-test * p-values* The Bonferroni Correction for Multiple Comparisons * ANOVA: Analysis of Variance* Count Data and the Chi-Square Statistic* Fitting a Line to Points on a Cartesian Plane* The R-Squared Coefficient of Determination* Correlation vs Causality*Segment 3: Predicting Outcomes with Regression** Features: Independent vs Dependent Variables* Linear Regression to Predict Continuous Values * Cost (or Loss) Functions * The Ordinary Least Squares Method* Logistic Regression to Predict Categories * Large Data Sets, Traditional Machine Learning, and Stochastic Gradient Descent* Overfitting* Regularization: L1 and L2* Deep Learning vs Frequentist Statistics* Resources for Further Study of Probability and Statistics  Segment 1: Frequentist Statistics<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
np.random.seed(42)<jupyter_output><empty_output><jupyter_text>Measures of Central Tendency Measures of central tendency provide a summary statistic on the center of a given distribution, a.k.a., the "average" value of the distribution.<jupyter_code>from scipy.stats import skewnorm
x = skewnorm.rvs(10, size=1000)
x[0:20]
fig, ax = plt.subplots()
_ = plt.hist(x, color = 'lightgray')<jupyter_output><empty_output><jupyter_text>Mean The most common measure of central tendency, synonomous with the term "average", is the **mean**, often symbolized with $\mu$ (population) or $\bar{x}$ (sample): $$ \bar{x} = \frac{\sum_{i=1}^n x_i}{n} $$<jupyter_code>xbar = np.mean(x)
xbar
fig, ax = plt.subplots()
plt.axvline(x = np.mean(x), color='orange')
_ = plt.hist(x, color = 'lightgray')<jupyter_output><empty_output><jupyter_text>Median The second most common measure of central tendency is the **median**, the midpoint value in the distribution:<jupyter_code>np.median(x)<jupyter_output><empty_output><jupyter_text>The mode is least impacted by skew, but is typically only applicable to discrete distributions. For continuous distributions with skew (e.g., salary data), median is typically the choice measure of central tendency:<jupyter_code>fig, ax = plt.subplots()
plt.axvline(x = np.mean(x), color='orange')
plt.axvline(x = np.median(x), color='green')
_ = plt.hist(x, color = 'lightgray')<jupyter_output><empty_output><jupyter_text>Measures of Dispersion  Variance $$ \sigma^2 = \frac{\sum_{i=1}^n (x_i-\bar{x})^2}{n} $$<jupyter_code>np.var(x)<jupyter_output><empty_output><jupyter_text>Standard Deviation A straightforward derivative of variance is **standard deviation** (denoted with $\sigma$), which is convenient because its units are on the same scale as the values in the distribution: $$ \sigma = \sqrt{\sigma^2} $$<jupyter_code>np.var(x)**(1/2)
sigma = np.std(x)
sigma
fig, ax = plt.subplots()
plt.axvline(x = xbar, color='orange')
plt.axvline(x = xbar+sigma, color='olivedrab')
plt.axvline(x = xbar-sigma, color='olivedrab')
_ = plt.hist(x, color = 'lightgray')<jupyter_output><empty_output><jupyter_text>Standard Error A further derivation of standard deviation is **standard error**, which is denoted with $\sigma_\bar{x}$:$$ \sigma_\bar{x} = \frac{\sigma}{\sqrt{n}} $$<jupyter_code>sigma/(len(x))**(1/2)
from scipy.stats import sem
sem(x) # defaults to 1 degree of freedom, which can be ignored with the larger data sets of ML
sem(x, ddof=0)<jupyter_output><empty_output><jupyter_text>Standard error enables us to compare whether the means of two distributions differ *significantly*, a focus of *Intro to Stats*.  Gaussian Distribution After <NAME>. Also known as **normal distribution**:<jupyter_code>x = np.random.normal(size=10000)
sns.set_style('ticks')
_ = sns.distplot(x)<jupyter_output><empty_output><jupyter_text>When the normal distribution has a mean ($\mu$) of zero and standard deviation ($\sigma$) of one, as it does by default with the NumPy `normal()` method...<jupyter_code>np.mean(x)
np.std(x)<jupyter_output><empty_output><jupyter_text>...it is a **standard normal distribution** (a.k.a., standard Gaussian distribution). Normal distributions are by far the most common distribution in statistics and machine learning. They are typically the default option, particularly if you have limited information about the random process you're modeling, because: 1. Normal distributions assume the greatest possible uncertainty about the random variable they represent (relative to any other distribution of equivalent variance). Details of this are beyond the scope of this tutorial. 2. Simple and very complex random processes alike are, under all common conditions, normally distributed when we sample values from the process. Since we sample data for statistical and machine learning models alike, this so-called **central limit theorem** (covered next) is a critically important concept.   The Central Limit Theorem To develop a functional understanding of the CLT, let's sample some values from our normal distribution:<jupyter_code>x_sample = np.random.choice(x, size=10, replace=False)
x_sample<jupyter_output><empty_output><jupyter_text>The mean of a sample isn't always going to be close to zero with such a small sample:<jupyter_code>np.mean(x_sample)<jupyter_output><empty_output><jupyter_text>Let's define a function for generating **sampling distributions** of the mean of a given input distribution:<jupyter_code>def sample_mean_calculator(input_dist, sample_size, n_samples):
    sample_means = []
    for i in range(n_samples):
        sample = np.random.choice(input_dist, size=sample_size, replace=False)
        sample_mean = np.mean(sample)
        sample_means.append(sample_mean)
    return sample_means
sns.distplot(sample_mean_calculator(x, 10, 10), color='green')
_ = plt.xlim(-1.5, 1.5)<jupyter_output><empty_output><jupyter_text>The more samples we take, the more likely that the sampling distribution of the means will be normally distributed:<jupyter_code>sns.distplot(sample_mean_calculator(x, 10, 1000), color='green')
_ = plt.xlim(-1.5, 1.5)<jupyter_output><empty_output><jupyter_text>The larger the sample, the tighter the sample means will tend to be around the population mean:<jupyter_code>sns.distplot(sample_mean_calculator(x, 100, 1000), color='green')
_ = plt.xlim(-1.5, 1.5)
sns.distplot(sample_mean_calculator(x, 1000, 1000), color='green')
_ = plt.xlim(-1.5, 1.5)<jupyter_output><empty_output><jupyter_text>Sampling from a skewed distribution<jupyter_code>s = skewnorm.rvs(10, size=10000)
_ = sns.distplot(s)
_ = sns.distplot(sample_mean_calculator(s, 10, 1000), color='green')
_ = sns.distplot(sample_mean_calculator(s, 1000, 1000), color='green')<jupyter_output><empty_output><jupyter_text>Sampling from a multimodal distribution<jupyter_code>m = np.concatenate((np.random.normal(size=5000), np.random.normal(loc = 4.0, size=5000)))
_ = sns.distplot(m)
_ = sns.distplot(sample_mean_calculator(m, 1000, 1000), color='green')<jupyter_output><empty_output><jupyter_text>Sampling from uniform Even sampling from the highly non-normal uniform distribution, the sampling distribution comes out normal:<jupyter_code>u = np.random.uniform(size=10000)
_ = sns.distplot(u, kde=False)
_ = sns.distplot(sample_mean_calculator(u, 1000, 1000), color='green')<jupyter_output><empty_output><jupyter_text>Therefore, with large enough sample sizes, we can assume the sampling distribution of the means will be normally distributed, allowing us to apply statistical and ML models that are configured for normally distributed noise, which is often the default assumption.As an example, the "*t*-test" (covered shortly in *Intro to Stats*) allows us to infer whether two samples come from different populations (say, an experimental group that receives a treatment and a control group that receives a placebo). Thanks to the CLT, we can use this test even if we have no idea what the underlying distributions of the populations being tested are, which may be the case more frequently than not.   z-scores Assuming normally-distributed data, a z-score indicates how many standard deviations away from the mean a data point (say, $x_i$) is: $$ z = \frac{x_i-\mu}{\sigma} $$ For example, let's say you get 85% on a stats test. Sounds like a pretty good score and you did extremely well relative to your peers if the mean was 60% with a standard deviation of 10%:<jupyter_code>x_i = 85
mu = 60
sigma = 10
x = np.random.normal(mu, sigma, 10000)
sns.distplot(x, kde=False, color='gray')
ax.set_xlim(0, 100)
plt.axvline(mu, color='orange')
for v in [-3, -2, -1, 1, 2, 3]:
    plt.axvline(mu+v*sigma, color='olivedrab')
_ = plt.axvline(x_i, color='blue')<jupyter_output><empty_output><jupyter_text>Your z-score is 2.5 standard deviations above the mean:<jupyter_code>z = (x_i - mu)/sigma
z<jupyter_output><empty_output><jupyter_text>Or using our simulated class of 10k students:<jupyter_code>z = (x_i - np.mean(x))/np.std(x)
z<jupyter_output><empty_output><jupyter_text>Less than one percent of the class outperformed you:<jupyter_code>len(np.where(x > 85)[0])
100*69/10000
np.percentile(x, 99)<jupyter_output><empty_output><jupyter_text>In contrast, if the mean score of your peers is 90 and the standard deviation is 2:<jupyter_code>mu = 90
sigma = 2
y = np.random.normal(mu, sigma, 10000)
sns.distplot(y, kde=False, color='gray')
plt.axvline(mu, color='orange')
for v in [-3, -2, -1, 1, 2, 3]:
    plt.axvline(mu+v*sigma, color='olivedrab')
_ = plt.axvline(x_i, color='blue')<jupyter_output><empty_output><jupyter_text>Your z-score is 2.5 standard deviations *below* the mean (!):<jupyter_code>z = (x_i - mu)/sigma
z<jupyter_output><empty_output><jupyter_text>Or using our simulated class of 10k students:<jupyter_code>z = (x_i - np.mean(y))/np.std(y)
z<jupyter_output><empty_output><jupyter_text>In which case, over 99% of the class outperformed you:<jupyter_code>len(np.where(y > 85)[0])
100*9933/10000
np.percentile(y, 1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09884884884884886, 'fraction_numerical': 0.017934601267934603, 'mean_word_length': 6.183343319352906, 'pattern_counts': {'":': 0, '<': 106, '<?xml version=': 0, '>': 108, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '545315', 'license': 'MIT', 'n_tokens_mistral': 3850, 'n_tokens_neox': 3542, 'n_words': 1308, 'path': 'notebooks/6-statistics.ipynb', 'repo_name': 'MilaI3/ML-foundations', 'size': '12002'}