<jupyter_start><jupyter_text>Statistics and Probability<jupyter_code>%matplotlib inline 
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from scipy import stats
import seaborn as sns
import time
np.random.seed(seed=int(time.time()))

# Make plots larger
plt.rcParams['figure.figsize'] = (15, 9)<jupyter_output><empty_output><jupyter_text>ProbabilityThe [probability](https://en.wikipedia.org/wiki/Probability) of a specified event is measure of the chance or likelihood that it will occur. Probability is quantified as a number between $0$ and $1$, where, loosely speaking, $0$ indicates impossibility and $1$ indicates certainty. The higher the probability of an event, the more likely it is that the event will occur. A simple example is the tossing of a fair (unbiased) coin. Since the coin is fair, the two outcomes ("heads" and "tails") are both equally probable; the probability of "heads" equals the probability of "tails"; and since no other outcomes are possible, the probability of either "heads" or "tails" is 1/2 (which could also be written as 0.5 or 50%).One way to determine this is to do repeated trails and count the success rate over the total tries. Another view would be subjective in nature, in other words an educated guess. An example of subjective probability is asking Boston Red Sox fans,  about the chances of the Red Sox winning the World Series. While there is no counting of trails, fans might still reply in actual percentage terms, based on their opinion of the strength of the team relative to its compeitiion.One also may have a subjective assessment of something that can be tested via trails. Say somebody believes she is a 90% free throw shooter. Then as she shoots, that subjective assessment can become more objective by decreasing the assessment with each miss and increasing the assessment with each make until it becomes a objective estimate of her true free throw percentage.* Probability is expressed in numbers between 0 and 1.  * Probability = 0 means the event never happens.   * Probability = 1 means it always happens.    * The total probability of all possible event always sums to 1.       Events and OutcomesIn [probability theory](https://en.wikipedia.org/wiki/Probability_theory), an outcome is a possible result of an experiment.Each possible outcome of a particular experiment is unique, and different outcomes are mutually exclusive (only one outcome will occur on each trial of the experiment). All of the possible outcomes of an experiment form the elements of a sample space.* The result of an experiment is called an [outcome](https://en.wikipedia.org/wiki/Outcome_(probability)).  * An event is any particular outcome or group of outcomes.  * A simple event is an event that cannot be broken down further.   * The sample space is the set of all possible simple events.  For the experiment where we flip a coin twice, the four possible outcomes that make up our sample space are (H, T), (T, H), (T, T) and (H, H), where "H" represents a "heads", and "T" represents a "tails". Outcomes should not be confused with events, which are sets (or informally, "groups") of outcomes. For comparison, we could define an event to occur when "at least one 'heads'" is flipped in the experiment - that is, when the outcome contains at least one 'heads'. This event would contains all outcomes in the sample space except the element (T, T).** Sample Space **  * Coin Toss ={head,tail} * Two coins S = {HH, HT, TH, TT}* Inspecting a part ={good,bad}* Rolling a die S ={1,2,3,4,5,6}  ProbabilityThe classical definition or interpretation of [probability](https://en.wikipedia.org/wiki/Probability_theory) is identified with the works of <NAME> and <NAME>. As stated in Laplace's Théorie analytique des probabilités,_The probability of an event is the ratio of the number of cases favorable to it, to the number of all cases possible when nothing leads us to expect that any one of these cases should occur more than any other, which renders them, for us, equally possible._  For example, if the event is "occurrence of an even number when a die is rolled", the probability is given by $\tfrac{3}{6}=\tfrac{1}{2}$, since 3 faces out of the 6 have even numbers and each face has the same probability of appearing.**Modern definition**The modern definition starts with a [countable set](https://en.wikipedia.org/wiki/Countable_set) called the [sample space](https://en.wikipedia.org/wiki/Sample_space) , which relates to the set of all _possible outcomes_ in classical sense, denoted by$\Omega$. It is then assumed that for each element $x \in \Omega\,$, an intrinsic "probability" value $f(x)\,$ is attached, which satisfies the following properties:$$f(x)\in[0,1]\mbox{ for all }x\in \Omega\,;$$$$\sum_{x\in \Omega} f(x) = 1\,.$$That is, the probability function $f(x$) lies between zero and one for every value of $x$ in the sample space $\Omega$, and the sum of $f(x$) over all values $x$ in the sample space $\Omega$ is equal to 1. An [event (probability theory](https://en.wikipedia.org/wiki/Event_(probability_theory)) is defined as any [subset](https://en.wikipedia.org/wiki/Subset) $E\,$ of the sample space$\Omega\,$. The _probability_ of the event$E\,$ is defined as$$P(E)=\sum_{x\in E} f(x)\,.$$So, the probability of the entire sample space is 1, and the probability of the null event is 0.The function$f(x)\,$ mapping a point in the sample space to the "probability" value is called a [probability mass function](https://en.wikipedia.org/wiki/Probability_mass_function) abbreviated as _pmf_. The modern definition does not try to answer how probability mass functions are obtained; instead it builds a theory that assumes their existence.**Probability:** the number of ways that an event can occur divided by the total number of possible outcomes.   What's the probability of getting heads in a fair coin flip?<jupyter_code>1/2.0<jupyter_output><empty_output><jupyter_text>What's the probability of getting an even number on a standard six-sided die?<jupyter_code>3/6.0<jupyter_output><empty_output><jupyter_text>What's the probability an 'ace' from a standard deck of cards?A standard deck of 52 playing cards consists of four suits (hearts, spades, diamonds and clubs). Spades and clubs are black while hearts and diamonds are red. Each suit contains 13 cards, each of a different rank: an Ace (which in many games functions as both a low card and a high card),cards numbered 2 through 10, a Jack, a Queen and a King.<jupyter_code>4/52.0<jupyter_output><empty_output><jupyter_text>OddsThe odds for an event is the ratio of the number of ways the event can occur compared to the number of ways it does not occur.For example, using the same events as above, the odds for:drawing a red card from a standard deck of cards is 1:1; anddrawing a club from that deck is 1:3.  What's the odds of a fair coin flip?<jupyter_code>1/1.0<jupyter_output><empty_output><jupyter_text>What's the odds of getting an even number on a standard six-sided die?<jupyter_code>3/3.0<jupyter_output><empty_output><jupyter_text>What's the odds of an 'ace' from a standard deck of cards<jupyter_code>4/48.0<jupyter_output><empty_output><jupyter_text>The Axioms of Probability First axiom - The probability of an event is a non-negative real number:$$P(E)\in\mathbb{R}, P(E)\geq 0 \qquad \forall E\in F$$where $F$ is the event space Second axiom -  unit measure:The probability that some elementary event in the entire sample space will occur is 1.$$P(\Omega) = 1.$$   Third axiom - the assumption of $\sigma$-additivity:Any countable sequence of disjoint (synonymous with mutually exclusive) events $E_1, E_2, ...$ satisfies$$P\left(\bigcup_{i = 1}^\infty E_i\right) = \sum_{i=1}^\infty P(E_i).$$The total probability of all possible event always sums to 1.  Consequences of these axiomsThe probability of the empty set:$$P(\varnothing)=0.$$Monotonicity   $$\quad\text{if}\quad A\subseteq B\quad\text{then}\quad P(A)\leq P(B).$$The numeric bound between 0 and 1:  $$0\leq P(E)\leq 1\qquad \forall E\in F.$$![Probability is expressed in numbers between 0 and 1](http://nikbearbrown.com/YouTube/MachineLearning/M10/Probability_0_1.png)    *Probability is expressed in numbers between 0 and 1.*   Probabilty of a certain event is 1:$$P(True) = 1$$Probability = 1 means it always happens.Probabilty of an impossible event is 0:$$P(False) = 0$$Probability = 0 means the event never happens.  Probabilty of A or B:$$P(A \quad or \quad B) = P(A) + P(B) - P(A \quad and \quad B) $$or$$P(A \cup B) = P(A) + P(A) - P(A \cap B) $$Probabilty of not A:$$P(not  \quad A) = 1- P(A) $$  If you pull a random card from a deck of playing cards, what is the probability it is a heart?<jupyter_code>13/52.0<jupyter_output><empty_output><jupyter_text>If you pull a random card from a deck of playing cards, what is the probability it is not a heart?<jupyter_code>1-(13/52.0)<jupyter_output><empty_output><jupyter_text>Independent EventsEvents A and B are independent events if the probability of Event B occurring is the same whether or not Event A occurs.Are the following independent events?* A fair coin is tossed two times. The two events are (1) first toss is a head and (2) second toss is a head.* You draw a card from a deck two times without replacement. The two events are (1) first draw is a heart and (2) second toss is a draw is a heart.  Probabilty of A or B The probability of either A or B occurring (or both) is$$P(A \quad or \quad B) = P(A) + P(B) - P(A \quad and \quad B) $$  Suppose we draw one card from a standard deck. What is the probability that we get a Queen or a King?$$P(King \quad or \quad Queen) = P(King) + P(Queen) - P(King \quad and \quad Queen) $$<jupyter_code>4/52.0+4/52.0-0<jupyter_output><empty_output><jupyter_text>Conditional ProbabilityIn probability theory, a [conditional probability](https://en.wikipedia.org/wiki/Conditional_probability) measures the probability of an event given that another event has occurred. That is,  "the conditional probability of A given B."    the conditional probability of A given B is defined as the quotient of the probability of the joint of events A and B, and the probability of B: $$ P(A|B) = \frac{P(A \cap B)}{P(B)}$$This may be visualized using a Venn diagram. ![P(A and B) Venn](http://nikbearbrown.com/YouTube/MachineLearning/M10/Conditional_Probability_Venn_Diagram.png)     *$P(A \cap B)$* Corollary of Conditional Probability is The Chain RuleIf we multiply both sides by $P(B)$ then$$ P(A|B) = \frac{P(A \cap B)}{P(B)}$$becomes$$ P(A|B) P(B) = P(A \cap B) $$If Events A and B are not independent, then$$P(A \quad and \quad B) = P(A) · P(B | A)$$ Statistical independenceEvents A and B are defined to be statistically independent if:$$\begin{align}             P(A \cap B) &= P(A) P(B) \\  \Leftrightarrow P(A|B) &= P(A) \\  \Leftrightarrow P(B|A) &= P(B)\end{align}$$That is, the occurrence of A does not affect the probability of B, and vice versaProbabilty of A or B for independent events $P(A and B) is 0$:$$P(A \quad or \quad B) = P(A) + P(A) $$  What is the probability that two cards drawn at random from a deck of playing cards will both be aces?<jupyter_code>(4/52.0)*(3/51.0)<jupyter_output><empty_output><jupyter_text>If you pull 2 cards out of a deck, what is the probability that both are hearts?If Events A and B are not independent, then$$P(A \quad and \quad B) = P(A) · P(B | A)$$$$P(Heart \quad and \quad Heart) = P(Heart) · P(Heart | Heart)$$<jupyter_code>(13/52.0)*(12/51.0)<jupyter_output><empty_output><jupyter_text>Bayes Rule[Bayes' theorem](https://en.wikipedia.org/wiki/Bayes%27_theorem) (alternatively Bayes' law or Bayes' rule) describes the probability of an event, given prior events. That is, a conditional probability.$$P(A|B) = \frac{P(A)\, P(B | A)}{P(B)},$$where A and B are events.* P(A) and P(B) are the independent probabilities of A and B.  * P(A | B), a conditional probability, is the probability of observing event A given that B is true.  * P(B | A), is the probability of observing event B given that A is true.   Bayesian inference[Bayesian inference](https://en.wikipedia.org/wiki/Bayesian_inference) is a method of statistical inference in which Bayes' theorem is used to update the probability for a hypothesis as evidence. Bayesian inference derives the posterior probability as a consequence of two antecedents, a prior probability and a "likelihood function" derived from a statistical model for the observed data. Bayesian inference computes the posterior probability according to Bayes' theorem:$$P(H\mid E) = \frac{P(E\mid H) \cdot P(H)}{P(E)}$$where,    $P(H\mid E)$ the posterior probability, denotes a conditional probability of $\textstyle H$ (the hypothesis) whose probability may be affected by the evidence $\textstyle E$.   $\textstyle P(H)$, the prior probability, is an estimate of the probability that a hypothesis is true, before observing the current evidence.   $\textstyle P(E\mid H)$ is the probability of observing $\textstyle E$ given $\textstyle H$. It indicates the compatibility of the evidence with the given hypothesis.   $\textstyle P(E)$ is sometimes termed the marginal likelihood or "model evidence". This factor is the same for all possible hypotheses being considered. Note that Bayes' rule can also be written as follows:$$P(H\mid E) = \frac{P(E\mid H)}{P(E)} \cdot P(H)$$where the factor $\textstyle \frac{P(E\mid H)}{P(E)}$ represents the impact of $E$ on the probability of $H$.     Bayesian probability exampleSuppose a certain disease has an incidence rate of 0.01% (that is, it afflicts 0.01% of the population).  A test has been devised to detect this disease.  The test does not produce false negatives (that is, anyone who has the disease will test positive for it), but the false positive rate is 1% (that is, about 1% of people who take the test will test positive, even though they do not have the disease).  Suppose a randomly selected person takes the test and tests positive.  What is the probability that this person actually has the disease?Bayes theorem would ask the question, what is the probability of disease given a postive result, or $P(disease\mid positive))$. What do we know?  $P(positive\mid disease)=1$ (i.e. The test does not produce false negatives.)     $P(disease)=0.0001$ (i.e.  1/10,000 have the disease)      $P(positive\mid no disease)=0.01$ (i.e. he false positive rate is 1%. This means 1% of people who take the test will test positive, even though they do not have the disease)      Bayes’ Theorem$$P(A|B) = \frac{P(A)\, P(B | A)}{P(B)},$$which can be rewritten as  $$P(A|B) = \frac{P(A)\, P(B | A)}{P(A)P(B|A)+P(\bar{A})P(B|\bar{A})},$$which in our example is$$P(disease|positive) = \frac{P(disease)\, P(positive | disease)}{P(disease)P(positive|disease)+P(no \quad  disease)P(positive|no \quad disease)},$$plugging in the numbers gives$$P(disease|positive)= \frac{(0.0001)\, (1)}{(0.0001)(1)+(0.9999)(0.01)}, \approx 0.01$$So even though the test is 99% accurate, of all people who test positive, over 99% do not have the disease.<jupyter_code>(0.0001*1)/((0.0001*1)+(0.9999*0.01))<jupyter_output><empty_output><jupyter_text>Bayesians versus Frequentists [Frequentist inference](https://en.wikipedia.org/wiki/Frequentist_inference) or frequentist statistics is a scheme for making statistical inference based on the frequency or proportion of the data. This effectively requires that conclusions should only be drawn with a set of repetitions.    Frequentists will only generate statistical inference given a large enough set of repetitions. In contrast, a Bayesian approach to inference does allow probabilities to be associated with unknown parameters.   ![Count Von Count](http://nikbearbrown.com/YouTube/MachineLearning/M10/Count_von_Count_kneeling.png)   *Count Von Count*   - from https://en.wikipedia.org/wiki/File:Count_von_Count_kneeling.png  While "probabilities" are involved in both approaches to inference, frequentist probability is essentially equivelent to counting. The Bayesian approach allows these estimates of probabilities to be based upon counting but also allows for subjective estimates (i.e. guesses) of prior probabilities.Bayesian probability, also called evidential probability, or subjectivist probability, can be assigned to any statement whatsoever, even when no random process is involved. Evidential probabilities are considered to be degrees of belief, and a Bayesian can even use an un-informative prior (also called a non-informative or Jeffreys prior).In Bayesian probability, the [Jeffreys prior](https://en.wikipedia.org/wiki/Jeffreys_prior), named after <NAME>, is a non-informative (objective) prior distribution for a parameter space.  The crucial idea behind the Jeffreys prior is the Jeffreys posterior. This posterior aims to reflect as best as possible the information about the parameters brought by the data, in effect  "representing ignorance" about the prior. This is sometimes called the "principle of indifference." Jeffreys prior is proportional to the square root of the determinant of the Fisher information:$$p\left(\vec\theta\right) \propto \sqrt{\det \mathcal{I}\left(\vec\theta\right)}.\,$$It has the key feature that it is invariant under reparameterization of the parameter vector $\vec\theta.$  At its essence the Bayesian can be vague or subjective about an inital guess at a prior probability. and the the posterior probability be updated data point by data point. A Bayesian defines a "probability" in the same way that many non-statisticians do - namely an indication of the plausibility or belief of a proposition.A Frequentist is someone that believes probabilities represent long run frequencies with which events occur; he or she will have a model (e.g. Guassian, uniform, etc.) of how the sample popluation was generated. The observed counts are considered a random sample the estimate the true parameters of the model.   It is important to note that most Frequentist methods have a Bayesian equivalent (that is, they give the same results) when there are enough repeated trails. They converge the the same result given enough data.    Probability distributions* We use probability distributions because they model data in real world.* They allow us to calculate what to expect and therefore understand what is unusual.* They also provide insight in to the process in which real world data may have been generated.* Many machine learning algorithms have assumptions based on certain probability distributions._Cumulative distribution function_A probability distribution Pr on the real line is determined by the probability of a scalar random variable X being in a half-open interval (-$\infty$, x], the probability distribution is completely characterized by its cumulative distribution function:$$ F(x) = \Pr[X \leq x] \quad \forall \quad x \in R .$$ Uniform Distribution$$X \equiv U[a,b]$$$$ f(x) = \frac{1}{b-a} \quad for \quad a \lt x \lt b$$$$ f(x) = 0 \quad for \quad a \leq x  \quad or  \quad \geq b$$$$ F(x) = \frac{x-a}{b-a} \quad for \quad a \leq x \lt b$$$$F(x) = 0 \quad for \quad x  \lt a  \quad  F(x) = 1 \quad for \quad x  \geq b$$![image Uniform Distribution"](http://172.16.58.3/YouTube/MachineLearning/M01/Uniform_Distribution_A.png)_Continuous Uniform Distribution_In probability theory and statistics, the continuous uniform distribution or rectangular distribution is a family of symmetric probability distributions such that for each member of the family, all intervals of the same length on the distribution's support are equally probable.- from [Uniform distribution (continuous  Wikipedia)](https://en.wikipedia.org/wiki/Uniform_distribution_(continuous))    ![image continuous  Uniform Distribution"](https://upload.wikimedia.org/wikipedia/commons/thumb/9/96/Uniform_Distribution_PDF_SVG.svg/375px-Uniform_Distribution_PDF_SVG.svg.png)![image continuous  Uniform Distribution"](https://upload.wikimedia.org/wikipedia/commons/thumb/6/63/Uniform_cdf.svg/375px-Uniform_cdf.svg.png)_Discrete Uniform Distribution_In probability theory and statistics, the discrete uniform distribution is a symmetric probability distribution whereby a finite number of values are equally likely to be observed; every one of n values has equal probability 1/n. Another way of saying "discrete uniform distribution" would be "a known, finite number of outcomes equally likely to happen".- from [Uniform distribution (discrete)  Wikipedia)](https://en.wikipedia.org/wiki/Uniform_distribution_(discrete))    ![image Uniform distribution (discrete) "](https://upload.wikimedia.org/wikipedia/commons/thumb/1/1f/Uniform_discrete_pmf_svg.svg/488px-Uniform_discrete_pmf_svg.svg.png)![imageUniform distribution (discrete) "](https://upload.wikimedia.org/wikipedia/commons/thumb/7/77/Dis_Uniform_distribution_CDF.svg/488px-Dis_Uniform_distribution_CDF.svg.png)<jupyter_code>#------------------------------------------------------------
# Define the distribution parameters to be plotted
W_values = [1.0, 3.0, 5.0]
linestyles = ['-', '--', ':']
mu = 0
x = np.linspace(-4, 4, 1000)

#------------------------------------------------------------
# plot the distributions
fig, ax = plt.subplots(figsize=(10, 5))

for W, ls in zip(W_values, linestyles):
    left = mu - 0.5 * W
    dist = stats.uniform(left, W)

    plt.plot(x, dist.pdf(x), ls=ls, c='black',
             label=r'$\mu=%i,\ W=%i$' % (mu, W))

plt.xlim(-4, 4)
plt.ylim(0, 1.2)

plt.xlabel('$x$')
plt.ylabel(r'$p(x|\mu, W)$')
plt.title('Uniform Distribution')

plt.legend()
plt.show()

# Adapted from http://www.astroml.org/book_figures/chapter3/fig_uniform_distribution.html<jupyter_output><empty_output><jupyter_text>Normal DistributionIn probability theory, the normal (or Gaussian) distribution is a very common continuous probability distribution. The normal distribution is remarkably useful because of the central limit theorem. In its most general form, under mild conditions, it states that averages of random variables independently drawn from independent distributions are normally distributed. Physical quantities that are expected to be the sum of many independent processes (such as measurement errors) often have distributions that are nearly normal. - from [Normal Distribution - Wikipedia)](https://en.wikipedia.org/wiki/Normal_distribution)   $$X \sim \quad N(\mu, \sigma^2)$$$$ f(x) = \frac{1}{\sigma \sqrt {2\pi }} e^{-\frac{( x - \mu)^2}{2\sigma^2}} \quad $$![image Normal Distribution"](http://172.16.58.3/YouTube/MachineLearning/M01/Normal_Distribution_A.png)![image Normal Distribution  "](https://upload.wikimedia.org/wikipedia/commons/thumb/7/74/Normal_Distribution_PDF.svg/525px-Normal_Distribution_PDF.svg.png)Normal cumulative distribution function![image Normal cumulative distribution function "](https://upload.wikimedia.org/wikipedia/commons/thumb/c/ca/Normal_Distribution_CDF.svg/525px-Normal_Distribution_CDF.svg.png)_Properties of normal distribution_- symmetrical, unimodal, and bell-shaped- on average, the error component will equal zero, the error above and below the mean will cancel out- Z-Score is a statistical measurement is (above/below) the mean of the data- important characteristics about z scores:  1. mean of z scores is 0  2. standard deviation of a standardized variable is always 1  3. the linear transformation does not change the _form_ of the distributionThe normal (or Gaussian) distribution was discovered in 1733 by <NAME> as an approximation to the binomial distribution when the number of trails is large.    ![image A<NAME> Moivre "](https://upload.wikimedia.org/wikipedia/commons/thumb/1/1b/Abraham_de_moivre.jpg/300px-Abraham_de_moivre.jpg)- from [Abraham de Moivre - Wikipedia)](https://en.wikipedia.org/wiki/Abraham_de_Moivre)The Gaussian distribution was derived in 1809 by <NAME>.    ![image <NAME> "](https://upload.wikimedia.org/wikipedia/commons/thumb/9/9b/Carl_Friedrich_Gauss.jpg/330px-Carl_Friedrich_Gauss.jpg)- from [<NAME> - Wikipedia)](https://en.wikipedia.org/wiki/Carl_Friedrich_Gauss)Importance lies in the Central Limit Theorem, which states that the sum of a large number of independent random variables (binomial, Poisson, etc.) will approximate a normal distribution Central Limit TheoremIn probability theory, the central limit theorem (CLT) states that, given certain conditions, the arithmetic mean of a sufficiently large number of iterates of independent random variables, each with a well-defined expected value and well-defined variance, will be approximately normally distributed, regardless of the underlying distribution. The central limit theorem has a number of variants. In its common form, the random variables must be identically distributed. - from [Central Limit Theorem - Wikipedia)](https://en.wikipedia.org/wiki/Central_limit_theorem)   The Central Limit Theorem tells us that when the sample size is large the average $\bar{Y}$ of a random sample follows a normal distribution centered at the population average $\mu_Y$ and with standard deviation equal to the population standard deviation $\sigma_Y$, divided by the square root of the sample size $N$. This means that if we subtract a constant from a random variable, the mean of the new random variable shifts by that constant. If $X$ is a random variable with mean $\mu$ and $a$ is a constant, the mean of $X - a$ is $\mu-a$. This property also holds for the spread, if $X$ is a random variable with mean $\mu$ and SD $\sigma$, and $a$ is a constant, then the mean and SD of $aX$ are $a \mu$ and $\|a\| \sigma$ respectively.This implies that if we take many samples of size $N$ then the quantity $$\frac{\bar{Y} - \mu}{\sigma_Y/\sqrt{N}}$$is approximated with a normal distribution centered at 0 and with standard deviation 1.  Central Limit Theorem in python<jupyter_code>x = np.arange(0, 7, 0.1)
dist = stats.uniform(1,5)
plt.plot(x, dist.pdf(x), lw=2)
print ("Mean:", dist.mean())
print ("Std Dev:", dist.std())
# Take a random sample of size 33
sample = dist.rvs(33)
print ("Sample Mean:", np.mean(sample))
print ("Sample Std Dev:", np.std(sample))
plt.hist(sample, bins=10)
plt.show()
# Take a random sample of size 3333
sample = dist.rvs(3333)
print ("Sample Mean:", np.mean(sample))
print ("Sample Std Dev:", np.std(sample))
plt.hist(sample, bins=10)
plt.show()
means = []
devs = []
samples = 10
for i in range(samples):
    sample = dist.rvs(33)
    means.append(np.mean(sample))
    devs.append(np.std(sample))
plt.hist(means, bins=20)
plt.title("Sample Means")
plt.show()

print ("Mean of Means:", np.mean(means))
print ("SD of Means:", np.std(means))
means = []
devs = []
samples = 100
for i in range(samples):
    sample = dist.rvs(33)
    means.append(np.mean(sample))
    devs.append(np.std(sample))
plt.hist(means, bins=20)
plt.title("Sample Means")
plt.show()

print ("Mean of Means:", np.mean(means))
print ("SD of Means:", np.std(means))
means = []
devs = []
samples = 1000
for i in range(samples):
    sample = dist.rvs(33)
    means.append(np.mean(sample))
    devs.append(np.std(sample))
plt.hist(means, bins=20)
plt.title("Sample Means")
plt.show()

print ("Mean of Means:", np.mean(means))
print ("SD of Means:", np.std(means))
means = []
devs = []
samples = 10000
for i in range(samples):
    sample = dist.rvs(33)
    means.append(np.mean(sample))
    devs.append(np.std(sample))
plt.hist(means, bins=20)
plt.title("Sample Means")
plt.show()

print ("Mean of Means:", np.mean(means))
print ("SD of Means:", np.std(means))
sns.distplot(dist.rvs(1000))
sns.plt.show()

sns.distplot(means)
sns.plt.show()<jupyter_output><empty_output><jupyter_text>Empirical distribution functions (ECDF)In statistics, an [empirical distribution function](https://en.wikipedia.org/wiki/Empirical_distribution_function) is the distribution function associated with the empirical measure of a sample. This cumulative distribution function is a step function that jumps up by 1/n at each of the n data points. Its value at any specified value of the measured variable is the fraction of observations of the measured variable that are less than or equal to the specified value.The empirical distribution function is an estimate of the cumulative distribution function that generated the points in the sample. It converges with probability 1 to that underlying distribution, according to the Glivenko–Cantelli theorem. A number of results exist to quantify the rate of convergence of the empirical distribution function to the underlying cumulative distribution function.![ECDF](http://nikbearbrown.com/YouTube/MachineLearning/DATA/Empirical_CDF.png)<jupyter_code>from statsmodels.distributions.empirical_distribution import ECDF

sample = np.random.uniform(0, 1, 10000)
ecdf = ECDF(sample)

x = np.linspace(min(sample), max(sample))
y = ecdf(x)
plt.step(x, y)
mu, sigma = 5, 3 # mean and standard deviation
sample = np.random.normal(mu, sigma, 10000)
ecdf = ECDF(sample)

x = np.linspace(min(sample), max(sample))
y = ecdf(x)
plt.step(x, y)<jupyter_output><empty_output><jupyter_text>Covariance[Covariance](https://en.wikipedia.org/wiki/Covariance) (How two variables depend on each other)  is a measure of the joint variability of two random variables. If the greater values of one variable mainly correspond with the greater values of the other variable, and the same holds for the lesser values, i.e., the variables tend to show similar behavior, the covariance is positive. In the opposite case, when the greater values of one variable mainly correspond to the lesser values of the other, i.e., the variables tend to show opposite behavior, the covariance is negative. The sign of the covariance therefore shows the tendency in the linear relationship between the variables. The magnitude of the covariance is not easy to interpret. The normalized version of the covariance, the [correlation coefficient](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient), however, shows by its magnitude the strength of the linear relation.The covariance between two jointly distributed real-valued random variable's $X$ and $Y$ with finite second moment's is defined as  the expected product of their deviations from their individual expected values:$$\operatorname{cov}(X,Y) = \operatorname{E}{\big[(X - \operatorname{E}[X])(Y - \operatorname{E}[Y])\big]}$$  Hypothesis testingA hypothesis is proposed for the statistical relationship between the two data sets, and this is compared as an alternative to an idealized null hypothesis that proposes no relationship between two data sets. The comparison is deemed statistically significant if the relationship between the data sets would be an unlikely realization of the null hypothesis according to a threshold probability—the significance level. Hypothesis tests are used in determining what outcomes of a study would lead to a rejection of the null hypothesis for a pre-specified level of significance. [Hypothesis tests](https://en.wikipedia.org/wiki/Statistical_hypothesis_testing) assume the thing you want to disprove, and then to look for evidence that the assumption is wrong. In this case, we assume that there is no difference between $\bar{x_1}$ and $\bar{x_2}$ (i.e. the mean of one distribution versus another) This is called the *null hypothesis* and is stated as$$H_0: \bar{x_1} = \bar{x_2}$$If $\bar{x_1}$ is very different from $\bar{x_2}$ we conclude that the null hypothesis is incorrect and that the evidence suggests there really is a difference between $\bar{x_1}$ and $\bar{x_2}$.There are many hypothesis tests that can be used to test whether there is a difference between $\bar{x_1}$ and $\bar{x_2}$:* Student’s T-Tests* One-Sample T-Test* Two-Sample T-Test* Paired T-Test* Wilcoxon Rank-Sum Test* Analysis of Variance (ANOVA)* Kruskal-Wallis TestWe will discuss these more in the module on hypothesis testing. P-valueTo determine how big the difference between  $\bar{x_1}$ and  $\bar{x_2}$ must be before we would reject the null hypothesis, we calculate the probability of obtaining a value of $\bar{x_2}$ as large as we have calculated if the null hypothesis were true. This probability is known as the *P-value*.In statistics, the p-value is a function of the observed sample results (a statistic) that is used for testing a statistical hypothesis. Before the test is performed, a threshold value is chosen, called the significance level of the test, traditionally 5% or 1% and denoted as $\alpha$.  If the p-value is equal to or smaller than the significance level ($\alpha$), it suggests that the observed data are inconsistent with the assumption that the null hypothesis is true and thus that hypothesis must be rejected (but this does not automatically mean the alternative hypothesis can be accepted as true). When the p-value is calculated correctly, such a test is guaranteed to control the Type I error rate to be no greater than $\alpha$.from [P-value](https://en.wikipedia.org/wiki/P-value)A is **p-value** is the probability of observing a test statistic equally or more extreme than the one you observed, assuming the hypothesis you are testing is true.     Confidence intervalsIn statistics, a confidence interval (CI) is a type of interval estimate of a population parameter. It provides an interval estimate for lower or upper confidence bounds. For $\beta_1$, usually referred to as a *confidence interval* and is typically +/-0.5% (a 99% confidence interval),+/-1% (a 98% confidence interval),+/-2.5% (a 95% confidence interval) or +/-5% (a 90% confidence interval). The lower and upper confidence bounds need not be equal, and they can be any number such that the confidence interval not exceed 100%.  The t-distributionIn probability and statistics, Student's t-distribution (or simply the t-distribution) is any member of a family of continuous probability distributions that arises when estimating the mean of a normally distributed population in situations where the sample size is small and population standard deviation is unknown. Whereas a normal distribution describes a full population, t-distributions describe samples drawn from a full population; accordingly, the t-distribution for each sample size is different, and the larger the sample, the more the distribution resembles a normal distribution.The t-distribution plays a role in a number of widely used statistical analyses, including the Student's t-test for assessing the statistical significance of the difference between two sample means, the construction of confidence intervals for the difference between two population means, and in linear regression analysis. The Student's t-distribution also arises in the Bayesian analysis of data from a normal family.- from [The t-distribution - Wikipedia)](https://en.wikipedia.org/wiki/Student%27s_t-distribution)When the CLT does not apply (i.e. as the number of samples is large), there is another option that does not rely on large samples When a the original population from which a random variable, say $Y$, is sampled is normally distributed with mean 0 then we can calculate the distribution of number of variants. In its common form, the random variables must be identically distributed. $$\sqrt{N} \frac{\bar{Y}}{s_Y}$$![image Student's t-distribution "](https://upload.wikimedia.org/wikipedia/commons/thumb/4/41/Student_t_pdf.svg/488px-Student_t_pdf.svg.png)Normal cumulative distribution function![image Normal cumulative Student's t-distribution "](https://upload.wikimedia.org/wikipedia/commons/thumb/e/e7/Student_t_cdf.svg/488px-Student_t_cdf.svg.png)  Which of distributions below are significantly different?<jupyter_code># Plot two normal distributions
domain = np.arange(-22, 33, 0.1)
values = stats.norm(3.3, 5.5).pdf(domain)
plt.plot(domain, values, color='r', linewidth=2)
plt.fill_between(domain, 0, values, color='#ffb6c1', alpha=0.3)
values = stats.norm(4.4, 6.6).pdf(domain)
plt.plot(domain, values, color='b', linewidth=2)
plt.fill_between(domain, 0, values, color='#89cff0', alpha=0.3)
plt.ylabel("Probability")
plt.title("Normal Distributions")
plt.show()
# Plot two normal distributions
domain = np.arange(1, 15, 0.1)
values = stats.norm(5.5, 1.1).pdf(domain)
plt.plot(domain, values, color='r', linewidth=2)
plt.fill_between(domain, 0, values, color='#ffb6c1', alpha=0.3)
values = stats.norm(9.9, 1.1).pdf(domain)
plt.plot(domain, values, color='b', linewidth=2)
plt.fill_between(domain, 0, values, color='#89cff0', alpha=0.3)
plt.ylabel("Probability")
plt.title("Normal Distributions")
plt.show()<jupyter_output><empty_output><jupyter_text>Statistical hypothesis testsThere are a few good statistical tests for hypothesis testing:* [ANOVA](https://en.wikipedia.org/wiki/Analysis_of_variance)* [Welch's t-test](https://en.wikipedia.org/wiki/Welch's_t-test)* [Mann-Whitney test](https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test)Each test makes various assumptions:* ANOVA assumes normal distributions and equal variances in the two data sets* The Welch t-test assumes normal distributions but not necessarily equal variances, and accounts for small sample sizes better* The Mann-Whitney test assumes nothing about the distributions but requires at least 20 data points in each set, and produces a weaker p-valueTypically you need to choose the most appropriate test. Tests that make more assumptions are more discriminating (stronger p-values) but can be misleading on data sets that don't satisfy the assumptions.All of these tests are available in the `scipy` library, a stats library for python:* [ANOVA](http://docs.scipy.org/doc/scipy-0.17.0/reference/generated/scipy.stats.f_oneway.html)* [Welch's t-test](http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_ind.html)* [Mann-Whitney](http://docs.scipy.org/doc/scipy-0.17.0/reference/generated/scipy.stats.mannwhitneyu.html)<jupyter_code># Generate some rnd_data_ and randomize

rnd_data_1 = []
rnd_data_1.extend(stats.norm(3.3, 5.5).rvs(3333))
np.random.shuffle(rnd_data_1)

rnd_data_2 = []
rnd_data_2.extend(stats.norm(4.4, 6.6).rvs(3333))
np.random.shuffle(rnd_data_2)

rnd_data_3 = []
rnd_data_3.extend(stats.norm(5.5, 1.1).rvs(3333))
np.random.shuffle(rnd_data_3)

rnd_data_4 = []
rnd_data_4.extend(stats.norm(9.9, 1.1).rvs(3333))
np.random.shuffle(rnd_data_4)

rnd_data_5 = []
rnd_data_5.extend(stats.norm(9.9, 1.1).rvs(3333))
np.random.shuffle(rnd_data_5)

# Make a rnd_data_ frame
rnd_data = pd.DataFrame()
rnd_data["A"] = rnd_data_1
rnd_data["B"] = rnd_data_2
rnd_data["C"] = rnd_data_3
rnd_data["D"] = rnd_data_4
rnd_data["E"] = rnd_data_5
rnd_data.head()
rnd_data.hist()
# To use ANOVA
statistic, pvalue = stats.f_oneway(rnd_data["A"], rnd_data["B"])
print (pvalue)
print (statistic)
# To use ANOVA
statistic, pvalue = stats.f_oneway(rnd_data["C"], rnd_data["D"])
print (pvalue)
print (statistic)
# To use ANOVA
statistic, pvalue = stats.f_oneway(rnd_data["A"], rnd_data["A"])
print (pvalue)
print (statistic)
# To use ANOVA
statistic, pvalue = stats.f_oneway(rnd_data["D"], rnd_data["E"])
print (pvalue)
print (statistic)
# to use Welch
statistic, pvalue = stats.ttest_ind(rnd_data["A"], rnd_data["B"])
print (pvalue)
print (statistic)
# to use Welch
statistic, pvalue = stats.ttest_ind(rnd_data["C"], rnd_data["D"])
print (pvalue)
print (statistic)
# to use Welch
statistic, pvalue = stats.ttest_ind(rnd_data["B"], rnd_data["B"])
print (pvalue)
print (statistic)
# to use Welch
statistic, pvalue = stats.ttest_ind(rnd_data["D"], rnd_data["E"])
print (pvalue)
print (statistic)
# Use the Mann-Whitney test on our data
# look up the function in scipy from the link above
# stats.mannwhitneyu
statistic, pvalue = stats.mannwhitneyu(rnd_data["A"], rnd_data["B"])
print (pvalue)
print (statistic)
# Use the Mann-Whitney test on our data
# look up the function in scipy from the link above
# stats.mannwhitneyu
statistic, pvalue = stats.mannwhitneyu(rnd_data["C"], rnd_data["D"])
print (pvalue)
print (statistic)
# Use the Mann-Whitney test on our data
# look up the function in scipy from the link above
# stats.mannwhitneyu
statistic, pvalue = stats.mannwhitneyu(rnd_data["C"], rnd_data["C"])
print (pvalue)
print (statistic)
# Use the Mann-Whitney test on our data
# look up the function in scipy from the link above
# stats.mannwhitneyu
statistic, pvalue = stats.mannwhitneyu(rnd_data["D"], rnd_data["E"])
print (pvalue)
print (statistic)
mw = stats.mannwhitneyu(rnd_data["A"], rnd_data["B"])
print (mw)
print (mw.statistic)<jupyter_output>MannwhitneyuResult(statistic=5177049.0, pvalue=7.7835551504176807e-07)
5177049.0
{'fraction_non_alphanumeric': 0.10241529190951298, 'fraction_numerical': 0.015163489084263558, 'mean_word_length': 4.968017686072218, 'pattern_counts': {'":': 0, '<': 80, '<?xml version=': 0, '>': 80, 'https://': 40, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '18283', 'license': 'MIT', 'n_tokens_mistral': 12194, 'n_tokens_neox': 11304, 'n_words': 4979, 'path': "Prof. Nik Brown's Notes/B01. Statistics and Probability/NBB_Statistics_and_Probability.ipynb", 'repo_name': 'PratikMahajan/Data-Science-Reference', 'size': '40588'}