<jupyter_start><jupyter_text>Tutorial 3: Probability<NAME>, 16 Oct 2019 $\newcommand{\E}{\mathrm{E}}$$\newcommand{\Var}{\mathrm{Var}}$$\newcommand{\Cov}{\mathrm{Cov}}$$\newcommand{\Corr}{\mathrm{Corr}}$This notebook introduces the elementary concepts from the probability theory that are necessary to understand and build machine learning models. For example, we will cover notions such as a random variable, a probability distribution and independence, as well as useful properties of probability like the chain rule or Bayes theorem. We will also illustrate these notions using Python code.   0. Preliminary steps<jupyter_code>import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
#%precision 3<jupyter_output><empty_output><jupyter_text>1. Definition of probabilityProbability, at its core, refers to the likelihood or the chance that a certain event will occur. For instance, when we roll a fair die, we can intuitively report that there is a probability of $\frac{1}{6}$ that one of the six possible outcomes (1, 2, 3, 4, 5 or 6) will occur. Probability, as a field, is branch of mathematics that provides tools to deal and measure uncertainty. In machine learning, we often have to deal with uncertain or stochastic quantities, whether the system that we try to model is assumed inherently stochastic like the trajectory of a subatomic particle, or cannot be completely observed like a robot with a low quality camera trying to identify objects, or because of incomplete modelling like when have to discard some of the information that we have collected (e.g., a robot that discretises the space when making a prediction about the position of an object). Before we move to more advanced notions, let's cover some elementary terminology that is commonly used in probability theory:  - **Experiment:** any process, usually random, that results in one of many possible results. For example in our first example, the toss of a die can be considered as an experiment.  - **Trial:** one repetition of the process of an experiment. For example, in the experiment of tossing a die 10 times and recording the sequence of results, each toss is considered as a trial. - **Outcome:** each distinct possible result of an experiment. For example, in the experiment of tossing a die twice, one possible outcome is (3,5) where 3 is the result from the first trial and 5 is the result from the second trial.  - **Event:** consists of one or more outcomes. For example, in the experiment of tossing on die, drawing an even number can be considered as an event, which has 3 possible outcomes (2, 4 and 6).- **Sample space** set of all possible outcome. For the die experiment, the sample space is {1,2,3,4,5,6}.In mathematical notation, the probability of an event E is often denoted as $P(E)$ or $Pr(E)$.  2. Random variablesA random variable is a function that maps a possible outcome of a random process to a real number. For example, we can define a random variable that represents the outcome from a die roll, $X \in \{1,2,3,4,5,6\}$, or the sum of the outcomes from rolling two dice, $Y \in \{1,2,\cdots,12\}$. Once we have defined a random variable, we can talk about its probability of taking a certain value or of taking a value in a given interval. Thus, the probability of observing a 3 when we roll a die is expressed as $P(X = 3)$, while the probability of observing an even outcome can be expressed as $P(X \in \{2,4,6\})$.   There are two types of random variables: discrete and continuous. A random variable is said to be discrete if it takes a finite or countably infinite number of values such as the variable $X$ representing the die roll's outcome. A continous random variable can take real values such as the value of a share in a stock exchange market.  3. Discrete variables 3.1. Probability mass functionThe probability mass function, or distribution of a discrete random variable $X$ maps the possible values of $X$ ($x_1, x_2, \dots$) to their probabilities. We will denote a probability mass function by a lower-cased $p(.)$: $p(x_i) = P(X = x_i)$ for all possible values $x_i$'s of X.For example, the probability mass function of the random variable $X$ that represents a die roll outcome is defined by: \begin{equation}p(x) = P(X = x) = \frac{1}{6} \;\; \text{for} \; x \in \{1, 2, 3, 4, 5, 6\}\end{equation}A probability mass function $p$ of a random variable $X$ must satisfy 3 conditions:1. The domain of $p$ must be the set of all possible outcomes of $X$.2. for all possible values $x$ of $X$, $0\leq p(x) \leq 1$, where $p(x) = 0$ if $x$ can never occur (impossible outcome) and $p(x) = 1$ if $x$ always occur.  3. $\sum_{x} {p(x)} = 1$, that is, the probabilities of all possible outcomes sum to 1.   3.2. Expectation and variance ExpectationThe expectation or expected value (or mean) of a discrete random variable $X$ is defined as:$\E(X) = \sum_{i} {x_ip(x_i)}$. In other words, the expectation is a weighted average of the possible values $x_i$ with weights $p(x_i)$, similar to a center of mass in physics. It gives us an idea about the central value taken by the random variable.For the die roll example above, the expectation is equal to:\begin{equation}\E(X) = 1 \cdot \frac{1}{6} + 2 \cdot \frac{1}{6} + 3 \cdot \frac{1}{6} + 4 \cdot \frac{1}{6} + 5 \cdot \frac{1}{6} + 6 \cdot \frac{1}{6} = \frac{21}{6} = 3.5\end{equation}The expectation has a few properties:- The expectation of a deterministic value is equal to the value itself: $\E(x) = x$, where x is a non-random value.- The expectation is a linear function, that is: $\E(aX + bY) = a\E(X) + b\E(Y)$.- $\E(f(X)) = \sum_{i} {f(x_i)p(x_i)}$ VarianceThe variance provides a measure of how much the values of a random variable $X$ vary as we sample different values of $X$ from its probability distribution. Mathematically, it is defined as: $\Var(X) = \E(\left[X - \E(X)\right]^2) = \E(X^2) - \E(X)^2$. For instance, the variance of $X$ in the die roll example is given by: \begin{equation}\Var(X) = \E(X^2) - \E(X)^2 = (1^2 \cdot \frac{1}{6} + 2^2 \cdot \frac{1}{6} + 3^2 \cdot \frac{1}{6} + 4^2 \cdot \frac{1}{6} + 5^2 \cdot \frac{1}{6} + 6^2 \cdot \frac{1}{6}) - \frac{21}{6} = \frac{71}{6} \end{equation}The variance has a few properties:- The variance of a deterministic value is equal to 0: $\Var(x) = 0$, where $x$ is a non-random value.- The variance preserves the sum: $\Var(X + Y) = \Var(X) + \Var(Y)$.- The variance is a non linear function for the multiplication by a scalar: $\Var(aX) = a^2\Var(X)$.- $\Var(f(X)) = \E(f(X)^2) - \E(f(X))^2$The standard deviation is a related measure defined as the square root of the variance: $SD(X) = \sqrt{\Var(X)}$.  3.3. Some popular discrete probability distributions   Bernoulli distribution Suppose that we run a random experiment whose outcome can be classified as either a success or a failure. An example of such an experiment is tossing a fair coin where a head is considered as success and tail as failure. If we let $X = 1$ when the outcome is a success (e.g. head) and $X = 0$ when it is a failure (e.g. tail), then the probability mass function of $X$ is given by: \begin{aligned}& p(0) = P(X = 0) = 1 − p\\& p(1) = P(X = 1) = p\end{aligned}where $p$ is the probability that the experiment is a success ($0\leq p\leq 1$).A random variable $X$ is said to be a Bernoulli random variable if its probability mass function is given by the equationsshown just above for some $p \in (0, 1)$. We also say that $X$ follows a Bernoulli distribution with parameter $p$ and write this mathematically as $X \sim Bernoulli(p)$.<jupyter_code># Mass function of a Bernoulli(0.7) distribution
print('p(0) ={p0: 1.1f}'.format(p0 = stats.bernoulli.pmf(0, p = 0.7)))
print('p(1) ={p0: 1.1f}'.format(p0 = stats.bernoulli.pmf(1, p = 0.7)))<jupyter_output>p(0) = 0.3
p(1) = 0.7<jupyter_text>We can draw samples from a Bernoulli distribution, using the function `stats.bernoulli.rvs()`. For example, if we want to draw 10 values from a $Bernoulli(0.7)$ distribution, we would use:<jupyter_code># Draw 10 values from a Bernoulli variable with p = 0.7
stats.bernoulli.rvs(p = 0.7, size = 10)<jupyter_output><empty_output><jupyter_text>The expection and variance of a random variable $X \sim Bernoulli(p)$ can be computed in a straightforward manner as:\begin{align}& \E(X) = 0 \cdot p(0) + 1 \cdot p(1) = p\\& \Var(X) = E(X^2) - E(X)^2 = (0^2 \cdot p(0) + 1^2 \cdot p(1)) - p^2 = p(1-p)\end{align}<jupyter_code># Expectation and variance of a X ~ Bernoulli(0.7)
print('E(X) ={exp: 1.2f}'.format(exp = stats.bernoulli.mean(p = 0.7)))
print('Var(X) ={var: 1.2f}'.format(var = stats.bernoulli.var(p = 0.7)))<jupyter_output><empty_output><jupyter_text>Binomial distribution Suppose now that we perform an experiment with $n$ independent trials, each of which results in a success with probability $p$ and in a failure with probability $1 − p$. If $X$ represents the number of successes that occur in the $n$ trials, then $X$ is said to be a binomial random variable with parameters $(n, p)$. A Bernoulli random variable is thus just a binomial random variable with parameters $(1, p)$. The probability mass function of a binomial random variable, $X$, having parameters $(n, p)$ (we can write $X \sim B(n,p)$), is given by:\begin{equation}p(i) = {n \choose i} p^i (1 − p)^{n−i} \quad i = 0, 1,\dots, n\end{equation}where ${n \choose i}$ is the number of possible combinations that result from randomly drawing $i$ objects out of $n$ different objects. Mathematically, ${n \choose i} = \frac{n!}{i!\,(n-i)!}$, where $k! = k\times(k-1)\times(k-2)\dots\times1$.<jupyter_code># Mass function of a B(6,0.5) distribution
for i in range(7):
    print('p({j:d}) = {p0:1.4f}'.format(j = i, p0 = stats.binom.pmf(i, n = 6, p = 0.5)))
# Plor of the mass function 
fig, ax = plt.subplots(1, 1)
n, p = 6, 0.5
x = list(range(7))
ax.plot(x, stats.binom.pmf(x, n, p), 'bo', ms=8, label='binom pmf')
ax.vlines(x, 0, stats.binom.pmf(x, n, p), colors='b', lw=5, alpha=0.5)
plt.show()<jupyter_output><empty_output><jupyter_text>In code, to draw samples from a Binomial distribution, we can call `stats.binom.rvs()`. For example, drawing 3 values from a $B(6, 0.5)$ distribution can be done using the following code:<jupyter_code># Draw 3 values from a variable following a B(6, 0.5) 
print(stats.binom.rvs(n = 6, p = 0.5, size = 3))<jupyter_output>[3 5 1]<jupyter_text>This means that we got 4 successes ($X = 1$) in each of the first two draws, and got 2 successes in the last draw. The expection and variance of a random variable $X \sim B(n,p)$ are given by:\begin{align}& \E(X) = np\\& \Var(X) = p(1-p)\end{align}<jupyter_code># Expectation and variance of a X ~ B(6, 0.5)
print('E(X) ={exp: 1.1f}'.format(exp = stats.binom.mean(n = 6, p = 0.5)))
print('Var(X) ={var: 1.1f}'.format(var = stats.binom.var(n = 6, p = 0.5)))<jupyter_output>E(X) = 3.0
Var(X) = 1.5<jupyter_text>4. Continuous variables  4.1. Probability density functionWhen working with a continuous random variable, we no longer want to find the probability that the variable take a specific value such as asking what is the probability that the value of a certain share in a financial market is exactly £100.684, since this probability will virtually be 0. Instead, we are more interested in the probability that the continuous random variable lies in a certain interval. For that, we normally describe the distribution of a random variable using a probability density function instead of a probability mass function. More specifically, a function $f$ is said to be the density of a random variable $X$ if, for any interval $I$, $P(X \in I) = \int_{I}{f(x)dx}$.In general, a function $f$ must satisfy 3 conditions to be considered as a density function of a random variable $X$:1. The domain of $f$ must be the set of all possible outcomes of $X$.2. for all possible values $x$ of $X$, $f(x) \geq 0$.3. $\int{f(x)dx} = 1$.  4.2. Expectation and variance ExpectationThe definition of expectation in the case of continuous random variables is very similar to that in the discrete case, where we only have to replace the summation with integration, and the probability mass function with the probability density function. More specifically, if $X\sim f$ (meaning that X follows a distribution whose density function is $f$) and the domain of $X$ is $\mathcal{X}$, then: $\E(X) = \int_{\mathcal{X}} {xf(x)dx}$.  VarianceFor variance, the definition remains the same: $\Var(X) = \E(\left[X - \E(X)\right]^2) = \E(X^2) - \E(X)^2$.  4.3. Cumulative distribution functionLet $X$ be a random variable. The function $F$ defined by: \begin{equation}F(x) = P(X \leq x), \quad -\infty \leq x\leq \infty\end{equation}is called the cumulative distribution function of $X$. Thus, $F$ specifies, for all real values $x$, the probability that the random variable is less than or equal to $x$.Similarly to the probability density function, the cumulative distribution function $F$ describes fully the distribution of a random variable, with the additional advantage that it can work with any type of random variables (discrete or continuous). Actually, F can be obtained from $f$, the probability density function, using the formulas: $F(x) = \int{f(t)dt}$ The One interesting property of $F$ is that it is a nondecreasing function; that is, if $a < b$, then $F(a) \leq F(b)$.  4.4. Some popular continuous probability distributions   Uniform distribution  A random variable $X$ is said to be uniformly distributed over the interval $(a, b)$, noted in short as $X \sim U(a,b)$, if $X$ is equally likely to fall anywhere in this interval. The probability density function of such random variable is given by:\begin{equation}f(x) =\left\{	\begin{array}{ll}		\frac{1}{b-a}  & \mbox{if } a\leq x\leq b \\		0 & \mbox{otherwise } 	\end{array}\right.\end{equation}<jupyter_code># density function of a U(0,1) distribution
print('f(-0.2) = {fx:1.1f}'.format(fx = stats.uniform.pdf(-0.2, loc = 0, scale = 1)))
print('f(0) = {fx:1.1f}'.format(fx = stats.uniform.pdf(0, loc = 0, scale = 1)))
print('f(0.3) = {fx:1.1f}'.format(fx = stats.uniform.pdf(0.1, loc = 0, scale = 1)))
print('f(1) = {fx:1.1f}'.format(fx = stats.uniform.pdf(1, loc = 0, scale = 1)))
print('f(4.5) = {fx:1.1f}'.format(fx = stats.uniform.pdf(4.5, loc = 0, scale = 1)))
# Plot of the density function 
fig, ax = plt.subplots(1, 1)
a, b = 0, 1
x = np.linspace(-2, 2, 200)
ax.plot(x, stats.uniform.pdf(x, loc = a, scale = b), 'r-', lw=2, label='frozen pdf')
plt.show()<jupyter_output><empty_output><jupyter_text>In code, to draw samples from a Uniform distribution, we can call `stats.uniform.rvs()`. For example, drawing 10 values from a $U(0, 1)$ distribution can be done using the following code:<jupyter_code># Draw 10 values from a variable following a U(0, 1) 
stats.uniform.rvs(loc = 0, scale = 1, size = 10)<jupyter_output><empty_output><jupyter_text>Since $F(x) = \int{f(t)dt}$, it follows from the above equation that the distribution function of a uniform random variable on the interval $(a, b)$ is given by:\begin{equation}F(x) =\left\{	\begin{array}{ll}		0  & \mbox{if } x< a \\		\frac{x-a}{b-a}  & \mbox{if } a\leq x\leq b \\        1 & \mbox{otherwise } 	\end{array}\right.\end{equation}The expectation and variance of the $U(a,b)$ distribution are given by:\begin{align}& \E(X) = \frac{a+b}{2}\\& \Var(X) = \frac{(b-a)^2}{12}\end{align}<jupyter_code># Expectation and variance of a X ~ U(0, 1)
print('E(X) ={exp: 1.1f}'.format(exp = stats.uniform.mean(loc = 0, scale = 1)))
print('Var(X) ={var: 1.4f}'.format(var = stats.uniform.var(loc = 0, scale = 1)))<jupyter_output>E(X) = 0.5
Var(X) = 0.0833<jupyter_text>Normal distribution$X$ is said to be a normal random variable (or normally distributed), with parameters $\mu$ and $\sigma^2$ if the density of $X$ is given by:\begin{equation}f(x) = \frac{1}{\sqrt{2\pi \sigma^2}} \exp\left(-\frac{1}{2 \sigma^2} (x-\mu)^2\right)\end{equation}In short, we denote a normal distribution with parameters $\mu$ and $\sigma^2$ by $\mathcal{N}(\mu,\sigma^2)$, so we will also write as before $X \sim \mathcal{N}(\mu,\sigma^2)$ to introduce a normal random variable $X$.<jupyter_code># density function of a N(0,1) distribution
print('f(-1.96) = {fx:1.2f}'.format(fx = stats.norm.pdf(-1.96, loc = 0, scale = 1)))
print('f(0) = {fx:1.2f}'.format(fx = stats.norm.pdf(0, loc = 0, scale = 1)))
print('f(1.96) = {fx:1.2f}'.format(fx = stats.norm.pdf(1.96, loc = 0, scale = 1)))
print('f(1) = {fx:1.2f}'.format(fx = stats.norm.pdf(1, loc = 0, scale = 1)))
# Plot of the density function 
fig, ax = plt.subplots(1, 1)
mu, sigma = 0, 1
x = np.linspace(-5, 5, 100)
ax.plot(x, stats.norm.pdf(x, loc = mu, scale = sigma), 'r-', lw=2, label='frozen pdf')
plt.show()<jupyter_output><empty_output><jupyter_text>In code, to draw samples from a Uniform distribution, we can call `stats.uniform.rvs()`. For example, drawing 10 values from a $U(0, 1)$ distribution can be done using the following code:<jupyter_code># Draw 10 values from a variable following a N(0, 1) 
stats.norm.rvs(loc = 0, scale = 1, size = 10)<jupyter_output><empty_output><jupyter_text>The expectation and variance of a normally distributed random variable, $X \sim \mathcal{N}(\mu,\sigma^2)$, are respectively equal to the paramters of the distribution, that is: $\E(X) = \mu$ and $\Var(X) = \sigma^2$. In other words a normal distribution is completely specified using its expectation and variance.<jupyter_code># Expectation and variance of a X ~ U(0, 1)
print('E(X) ={exp: 1.1f}'.format(exp = stats.norm.mean(loc = 0, scale = 1)))
print('Var(X) ={var: 1.1f}'.format(var = stats.norm.var(loc = 0, scale = 1)))<jupyter_output>E(X) = 0.0
Var(X) = 1.0
{'fraction_non_alphanumeric': 0.13502968617472436, 'fraction_numerical': 0.026576194515125814, 'mean_word_length': 4.428483732351135, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '40735', 'license': 'MIT', 'n_tokens_mistral': 5940, 'n_tokens_neox': 5634, 'n_words': 2454, 'path': 'Tutorial3_Probability/Probability.ipynb', 'repo_name': 'Adnane017/ML_tutorials', 'size': '17693'}