<jupyter_start><jupyter_text>Normal Distribution IntroductionThe first formal model in these notes happened so fast, you might have missed it.  By assuming $X_n \sim_{iid} \text{Bernoulli}(p)$, we created a single model.  This one statistical model assumed the Bernoulli distribution.  Our data consisted of multiple independent observations from the identical distribution (iid), a Bernoulli distribution with unknown population parameter $p$.In this section, we change the assumed distribution to the Normal distribution.  Because the support for the Normal distribution is all real numbers, this distribution applies to data that could potentially take on any value in the real line.  We complete the section by rehearsing our use of the likelihood function as it applies to Normal data.  When, out in the real world on your own, if don't know what model to apply, assume normality.   Warm Up- **numeric**: a type of variable that represents numbers, meaningfully mathable values.  For example, adult heights.- **continuous random variable**: a random variable that can take on any value within a given range.  For example, a random varible that measures time.- **density plot**: a visualization of an approximation of the probability density function.- **standardize**: a mathematical trick to make unitless a dataset.- **standard normal distribution**: a instance of the normal distribution where the mean is $0$ and the standard deviationis $1$.- **sampling distribution**: the name of the distribution of a statistic.- **Central Limit Theorem**: a theorem that says the statistic the mean looks normal if the sample size is big enough. Density PlotIt's common to assume any single numerical variable data follows a Normal distribution, $ Y_n \sim \text{Normal}(\mu, \sigma)$ for $n = 1, 2, \ldots, N$.  Note that we will call this a formal model although I understand that it is difficult to separate the ideas of distribution and model at this point.  For now, let's push forward and begin as we ought to begin any analysis, plot the data.<jupyter_code>import numpy as np
import pandas as pd
from scipy.stats import norm as normal
import bplot as bp

bp.LaTeX()
bp.dpi(300)<jupyter_output><empty_output><jupyter_text>Let's return to our dataset about the Order Carnivora.  Consider the variable `BW`, which records birth weight in grams.  These are numeric data and it's reasonable to assume the data came from a continuous random variable, because grams can theoretically take on any positive value in a reasonable range for birth weights from the Order Carnivora.  Since our dataset has missing values, recorded as `NaN`s, let's first drop them.  Next we'll approximate the density function from which these data may have come from.  A density plot is an approximation, from the available data, of the probability density function itself.  A common alternative plot is a histogram.<jupyter_code>carnivora = pd.read_csv("https://raw.githubusercontent.com/roualdes/data/master/carnivora.csv")
bw = carnivora['BW'].dropna()
bp.density(bw)
bp.histogram(bw)
bp.labels(x='Birth weight (g)', size=18)<jupyter_output><empty_output><jupyter_text>The above plot tells us that the majority of our data consist of observations below $500$ grams with a few observations showing up sporadically above $500$.  How many observations are above $500$ is not immediately clear, so let's see if we can modify the plot above to help us visualize all of the data.<jupyter_code>bw = carnivora['BW'].dropna()
bp.density(bw)
bp.rug(bw)
bp.labels(x='Birth weight (g)', size=18)<jupyter_output><empty_output><jupyter_text>Using vectorization, try to produce code that counts the number of observations greater than $500$.  And then turn this into a proportion.  What does this proportion represent, in terms of a Bernoulli distribution?  Properties of the Normal Distribution If $Y \sim \text{Normal}(\mu, \sigma)$, then one can standardize the random variable $Y$ by subtracting off the mean $\mu$ and scaling by the standard deviation $\sigma$,$$ Z = \frac{Y - \mu}{\sigma} \sim \text{Normal}(0, 1). $$This linear transformation of a random variable that follows a normal distribution is so common, that the random variable $Z$ has a special name.  A random variable that follows the $\text{Normal}(0, 1)$ distribution is called standard Normal; $Z$ follows a standard Normal distribution with mean $\mu = 0$ and variance $\sigma = 1$.The probability density function of the $\text{Normal}(\mu, \sigma)$ distribution is$$ \text{normal}(x | \mu, \sigma) = (2\pi\sigma^2)^{-1/2} \exp{\left( \frac{-(x - \mu)^2}{2\sigma^2} \right) }. $$A plot of the standard Normal probability density function is displayed below.  Try to change the code to help you better understand that the $\text{Normal}(\mu, \sigma)$ distribution indexes an uncountable number of distributions via $\mu$ and $\sigma$.  For each specific choice of $(\mu, \sigma)$, think of it as an instantiation of a new random variable.<jupyter_code>x = np.linspace(-4, 4, 101)
mu = 0; sigma = 1;
fx = normal.pdf(x, loc=mu, scale=sigma)
bp.curve(x, fx)
bp.labels(x='$x$', y='$f(x)$', size=18)<jupyter_output><empty_output><jupyter_text>Likelihood ExampleTo estimate the population mean birth weight of animals from the Order Carnivora $\mu$, we'll assume a Normal distribution, $Y_n \sim_{iid} \text{Normal}(\mu, \sigma)$.  Find the MLE of $\mu$ and then $\sigma$ by hand. ExampleFind the MLE of $(\mu, \sigma)$ using a computer.  Assumed NormalityNotice from the plot above of birth weights from the Order Carnivora, the data don't obviously come from a Normal distribution, and yet we modeled these data with a Normal distribution.  Such an assumption doesn't offend nearly any statistician, and yet it's almost offensive that no statistician is bothered by this.  In this subsection, we'll explore why statisticians are often happy to assume normality.Statisticians are not often bothered by assuming Normal data, because they are trained to not think about data statically.  The randomness of data, despite the data appearing to be fixed quantities, comes from imagining that the process that produced these data could be repeated (even if it can't).  In the case of a random sample of animals from the Order Carnivora, this would mean that you could (but wouldn't) randomly sample a new set of data from the same population of animals from the Order Carnivora.Recall that earlier in these notes, we already saw this idea.  Our operational definition of probability is the limitting relative frequency of repeating the process an infinite number of times.  We are now expanding on this idea to imagine that an entire new dataset comes from each iteration.Statisticians recognize that collecting new data is unlikely to happen, but based on our operational definition of probability, this theoretical resampling is just the natural, logical extension.  Since each new dataset comes from the same population, we'd assume the Normal distribution for each new dataset.  Based on the model $Y_n \sim \text{Normal}(\mu, \sigma)$, the likelihood dictates that we use the sample mean to estimate $\mu$.  The punchline to all of this is that statisticians can prove mathematically the shape of the multiple estimates of $\mu$ that would come about based on this infinite resampling.Just like we can use a computer to approximate the probability a coin flip turning up heads to be $1/2$, we can simulate the shape of multiple esimates of $\mu$ from a population of animals from the Order Carnivora.  Let's look at the code and a plot, and then I'll explain what's going on.<jupyter_code>N = bw.size
R = 1001
mus = np.full((1001, ), np.nan)

seed = np.random.seed(1234)
for r in range(R):
    idx = np.random.choice(bw._index, N)
    mus[r] = bw[idx].mean()
    
bp.density(mus)
bp.rug(mus)

x = np.linspace(np.min(mus), np.max(mus), 101)
fx = normal.pdf(x, loc=bw.mean(), scale=bw.std()/np.sqrt(N))

bp.curve(x, fx, color='tab:orange')
bp.labels(x='Mean birth weight (g)', y='Density', size=18)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08091337800943162, 'fraction_numerical': 0.007942417473318442, 'mean_word_length': 4.8229768786127165, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '387862', 'license': 'MIT', 'n_tokens_mistral': 2167, 'n_tokens_neox': 2058, 'n_words': 1130, 'path': 'docs/content/normal.ipynb', 'repo_name': 'josephmlev/314', 'size': '8058'}