<jupyter_start><jupyter_text>The Normal Distribution* The `normal` or `Gaussian` distribution is a continuous probability distribution.  * Numerous real-world phenomena can be modeled, or approximated using normal distributions.* Some examples:  * The heights of males in a population  * Errors in an instrument  * Duration of cab rides    * Many statistical tests assume that your data is distributed according to a normal distribution  * More on this later  Properties of the Normal Distribution* The normal distribution is a continuous distribution  * Typically graphically represented as a continuous curve,     * As opposed to the discrete distribution, the continuous can usually take any values either in a closed or open range  * The probability space is uncountably infinite* The Gaussian is bell-shaped  * Symmetric with a mode in middle  Properties of the Normal Distribution - Cont'd- The `parameters` for a normal distribution are the location of its center (mean) and spread (standard deviation)  - $\mu$ is the mean; it governs where the middle of the distribution lies on the $x$-axis  - $\sigma$ is the standard deviation; determines how wide the bell is and can also be referred to as the scale- To describe that a random variable is distributed according to a normal with parameters $\mu$ and $\sigma$, we write:$$X \sim \mathcal{N} (\mu,\sigma)$$As with the binomial, two gussian distributions with different parameters tell different stories:$$X \sim (\mu = 0, \sigma = 1) \ne Y \sim (\mu = 1, \sigma = 1) \ne Z \sim (\mu = 2, \sigma = 2) \ne$$<jupyter_code>plt.figure(figsize=(12, 4))

x = np.arange(-3.5, 3.5, 0.05)
p_x = [norm.pdf(val, 0, 1) for val in x]
plt.plot(x, p_x, label="$\mu = 0 $ and $\sigma = 1$")

x = np.arange(-13, 17, 0.05)
p_x = [norm.pdf(val, 2, 5) for val in x]
plt.plot(x, p_x, label="$\mu = 2 $ and $\sigma = 5$", color='r')
plt.legend()<jupyter_output><empty_output><jupyter_text>The Spread and Probability- For the blue curve ($\mu = 0$ and $ \sigma = 1$), the range of values with high probabilities is small    - For the red curve ($\mu = 2$ and $\sigma = 5$), the range of values with low probabilities is larger than that of the blue curve ($\mu = 0$ and $\sigma = 0.2$)- Think of this in terms of the probability distribution of the biased die form the binomial example.  - If you were to draw a random value -- just like when we were sampling from the die -- what do you think the values obtained from the red distribution would look like?  Properties of the Normal Distribution- The probability density function for a random variable $X \sim \mathcal{N} (\mu,\sigma)$ is:$$     p(x) = \frac{1}{\sqrt{2\pi\sigma^2}} e^{\frac{(x-u)^2}{2\sigma^2}}$$- Since the sample space is uncountably infinite, the probability of each event is 0- Therefore, instead of computing the `pmf`, continuous random variables compute the `pdf`  `pdf` versus `pmf`- A `pdf` is conceptually similar to a `pmf`, but instead of giving you a probability, it tells you the density around a particular point.   - You can think of it as mass (as in `pmf`) normalized by a small area      - ex. in the  $x \sim \mathcal{N}(0, 1)$ probability density around $0$ is much higher than the density around $-2.5$- Since $X$ is continuous, the `pdf` is also continuous- The total area under the `pdf` curve sums to 1  Data in the distribution - 1 $\sigma$- In the normal distribution, the area with the highest density lies close to the mean  - Thu, when we sample from a normal, the bulk of the observations generated from a normal distribution lie near the mean          Data in the distribution - 2 $\sigma$  Data in the distribution - 3 $\sigma$<jupyter_code># We manually implement the CDF
def computePDF_normal (x, mu, sigma):
    return 1.0/(sigma * np.sqrt(2 * np.pi)) * np.exp( - (x - mu)**2.0 / (2 * sigma**2))


# We manually implement the CDF
def computePDF_normal (x, mu, sigma):
    return 1.0/(sigma * np.sqrt(2 * np.pi)) * np.exp( - (x - mu)**2.0 / (2 * sigma**2))
    

f, (ax1, ax2, ax3) = plt.subplots(3)
f.set_figheight(15)
f.set_figwidth(15)
x_values = np.array(np.arange(-6.5,2,0.01))
ax1.fill_between(x=x_values, 
                 y1=computePDF_normal(x_values, 0, 2) ,
                 facecolor="blue",
                 alpha=0.35)

x_values = np.arange(2,6.5,0.01)
ax1.fill_between(x=x_values, 
                 y1= computePDF_normal(x_values, 0, 2),
                 facecolor="blue",
                 alpha=0.35)
x_values = np.arange(-2,2,0.01)
ax1.fill_between(x=x_values, 
                 y1= computePDF_normal(x_values, 0, 2),
                 facecolor='red',
                 alpha=0.35)
ax1.text(x=-2.8, y=0.03, s= "13%")
ax1.text(x=-0.1, y=0.1, s= "68%")
ax1.text(x=2.8, y=0.03, s="13%")





x_values = np.array(np.arange(-6.5,4,0.01))
#print computePDF(x_values, 0, 0.1)
ax2.fill_between(x=x_values, 
                 y1=computePDF_normal(x_values, 0, 2) ,
                 facecolor="blue",
                 alpha=0.35)

x_values = np.arange(4,6.5,0.01)
ax2.fill_between(x=x_values, 
                 y1= computePDF_normal(x_values, 0, 2),
                 facecolor="blue",
                 alpha=0.35)
x_values = np.arange(-4,4,0.01)
ax2.fill_between(x=x_values, 
                 y1= computePDF_normal(x_values, 0, 2),
                 facecolor='red',
                 alpha=0.35)

ax2.text(x=-4.8, y=0.03, s= "2.5%")
ax2.text(x=-0.1, y=0.1, s= "95%")
ax2.text(x=4.8, y=0.03, s="2.5%")


x_values = np.array(np.arange(-6.5,6,0.01))
ax3.fill_between(x=x_values, 
                 y1=computePDF_normal(x_values, 0, 2) ,
                 facecolor="blue",
                 alpha=0.35)

x_values = np.arange(6,6.5,0.01)
ax3.fill_between(x=x_values, 
                 y1= computePDF_normal(x_values, 0, 2),
                 facecolor="blue",
                 alpha=0.35)
x_values = np.arange(-6,6,0.01)
ax3.fill_between(x=x_values, 
                 y1= computePDF_normal(x_values, 0, 2),
                 facecolor='red',
                 alpha=0.35)
ax3.text(x=-6.8, y=0.03, s= "1%")
ax3.text(x=-0.1, y=0.1, s= "99%")
ax3.text(x=6.8, y=0.03, s="1%")<jupyter_output><empty_output><jupyter_text>Computing the `pdf` in Python* We can easily compute the `pdf` in python using `scipy````pythonimport scipy.statsnorm_0_1 = scipy.stats.norm(0, 1)print(norm_0_1.pdf(2))```* `pdf` can also take a range of values```pythonprint(norm_0_1.pdf([-2, -1, 0, 1, 2]))```* `scipy.stats.norm` creates a frozen distribution  * This can be thought of a constructor in OO programming  * Can also invoke pdf using static methods `scipy.stats.norm.pdf(0, loc=0, scale=1)`* Which of the pdf values is larger and why? * Which of the `pdf`s is largest `pdf(-2)` or `pdf(2)`?<jupyter_code>norm_0_1.rvs(10)
import scipy.stats
norm_0_1 = scipy.stats.norm(0, 1)
print(norm_0_1.rvs(3)) # return 3 random variates form Normal with mean 0 and std. dev = 1
print(norm_0_1.pdf(11)) # compute the pdf of 11 according to a Normal with mean 0 and std. dev = 1
print(norm_0_1.pdf([11, 2])) # compute the pdf of 2 according to a Normal with mean 0 and std. dev = 1
scipy.stats.norm.rvs
scipy.stats.norm.rvs?
print(scipy.stats.norm.rvs(loc=0, scale=1, size=3))
print(scipy.stats.norm.pdf(11, loc=0, scale=1))
print(scipy.stats.norm.pdf([11,2], loc=0, scale=1))
print(norm_0_1.pdf([-2, -1, 0, 1, 2]))<jupyter_output>[0.05399097 0.24197072 0.39894228 0.24197072 0.05399097]<jupyter_text>The Sum of PDF > 1```python >>> print(sum(norm_0_1.pdf([-2, -1, -0.5, 0, 0.5, 1, 2])))1.6949963159946946```* Why is the sum of the pdfs for the small subsample of the sample space greater than 1?<jupyter_code>print(norm_0_1.pdf(11))
print(computePDF_normal(11, 0, 1))


print("*" * 25)
print(norm_0_1.pdf([-2, -1, 0, 1, 2]))
print([float(format(computePDF_normal(x, 0, 1), ".8f")) for x in [-2, -1, 0, 1, 2]])<jupyter_output>2.1188192535093538e-27
2.1188192535093538e-27
*************************
[0.05399097 0.24197072 0.39894228 0.24197072 0.05399097]
[0.05399097, 0.24197072, 0.39894228, 0.24197072, 0.05399097]<jupyter_text>Plotting the `pdf`* We cannot compute the `pdf` for all the possible values  * Infinite number of values  * We can compute it for a sufficiently large number of values such that the plot looks continuous and smooth.* we can use `np.arange` to generate values in a range with a specific step value```pythonnp.arange(start, stop, step)```  * As with Python ranges, stop is not included in the range<jupyter_code>np.arange(-2, 2, 0.5)
plt.figure(figsize=(10,4))

norm_0_1 = scipy.stats.norm(0, 1)
x_axis = np.arange(-2, 2.5, 0.5)
y_axis = norm_0_1.pdf(x_axis)
plt.plot(x_axis, y_axis)

plt.figure(figsize=(10,4))

norm_0_1 = scipy.stats.norm(0, 1)
x_axis = np.arange(-4, 4.05, 0.05)
y_axis = norm_0_1.pdf(x_axis)

plt.plot(x_axis, y_axis)<jupyter_output><empty_output><jupyter_text>Illustrating Density* To illustrate the idea that dense regions contribute more random samples, we sample $1000$ values from a standard normal distribution $\mathcal{N}(\mu=0, \sigma=1)$* We will plot the values as a histogram  * We bin the values into 20 bins and plot them as a histogram  * Counts for each bin are shown on the $y-$axis<jupyter_code># Ex. sampling 10 values from Gaussian with mean 0 and standard deviation 2
np.random.seed(225)
gaussianSamples = np.random.normal(0, 2, 10)
gaussianSamples

np.random.seed(225)
gaussianSamples = np.random.normal(0, 2, 1000)
gaussianSamples[0:10]

# plot the counts, rathe than the density
plt.figure(figsize=(10, 4))
_ = plt.hist(gaussianSamples, bins = 20, edgecolor="k", linewidth=2, alpha=0.4)
# plots the density rather than the counts
plt.figure(figsize=(10, 4))
_ = plt.hist(gaussianSamples, bins = 20, edgecolor="k", linewidth=2, alpha=0.4, density=True)<jupyter_output><empty_output><jupyter_text>Comparing the Density Curve and the Histogram- We plot the pdf over the histogram to see if it matches the sequence data- Should the histogram match the density curve exactly?<jupyter_code>plt.figure(figsize=(8, 4))

normal_0_2 = scipy.stats.norm(0, 2)
x_axis = np.linspace(-8, 8, 100)
y_axis = normal_0_2.pdf(x_axis)

plt.plot(x_axis, y_axis, color="k")

# Note that the param density=True is necessary here
_ = plt.hist(gaussianSamples, bins = 20, edgecolor="k", color='#80d4ff', linewidth=2, alpha=0.2, density=True)


from scipy.stats import norm
# we don't need to create the `frozen` distribution beforehand
norm.pdf(0, loc=0, scale=2) # equivalent to norm.pdf(0, 0, 2)

some_vals = [-2, -1, -0.5, -0.2, -0,1, 0, 0.1, 0.2, 0.5, 1, 2 ]

mu = 0
sigma = 2

print(f"For mu={mu} and sigma={sigma}:")


for val in some_vals:
    print(f"\tThe probability density of x= {val} is {norm.pdf(val, 0, 2)}")

print("The sum of the probility densities computed above is: ", end="\t")
print(sum([norm.pdf(x, 0, 2) for x in some_vals]))<jupyter_output>The sum of the probility densities computed above is: 	2.1518536163942685<jupyter_text>The sum of `pdf`s- Why is the sum here greater than 0? Aren't the probabilities of the elements in the sample space supposed to sum to 1?- Remember that the probability at a single point is 0  * The sample space is infinite  - Therefore, `pdf` computes the probability of small interval. (ex. $x-0.01 < x <  x+0.01$)* We know that:$$density = \frac{mass}{area}$$So, we can convert back to mass such that$$mass = density * area$$<jupyter_code>norm_cdf_sum = 0
for x in np.arange(-8, 8, 0.0001):
    norm_cdf_sum +=  computePDF_normal(x, 0, 2) * 0.0001
print("The sum of the probility densities computer above is: %s" % norm_cdf_sum, end="\t")<jupyter_output>The sum of the probility densities computer above is: 0.9999366575184352
{'fraction_non_alphanumeric': 0.135863018413354, 'fraction_numerical': 0.0641025641025641, 'mean_word_length': 3.765477654776548, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '55032', 'license': 'Apache-2.0', 'n_tokens_mistral': 4507, 'n_tokens_neox': 4056, 'n_words': 1353, 'path': 'morea/intro_probability/resources/15_probability_distributions_gaussian.ipynb', 'repo_name': 'mahdi-b/ICS434-Sp2022', 'size': '11622'}