<jupyter_start><jupyter_text>**Artificial Intelligence - MSc**CS6462 - PROBABILISTIC AND EXPLAINABLE AI SEM2 2021/2CS6462_Lab_1.3<NAME>  Probabilistic Modelling* A model is a mathematical description of a phenomenon.* A probabilistic model is a probabilistic description of a phenomenon  that models our **uncertainty** about it.* Mathematically, a probabilistic model is a (set of) probabilistic distribution(s)You can express it *  either writing down its analytical formula *  or plotting its probability mass function  Generative model* A probabilistic model is a generative model.* It means that you can use it to simulate a phenomenon (by *sampling* from the model)What is a sample?* A sample is an instance of the simulation Queries Given a probabilistic model, we want to  query the model* Will it rain tomorrow?* Given that today temperature is 15°C, will it rain tomorrow?We can answer this type of questions * using probability calculus (marginalisation,  Bayes'rule, expectations)However, we will see for the case of continuous variables that* answering these questions requires solving difficult integralsInstead, we will see that by * *sampling the model* we can get an approximate answer to our questions without the need of computing those integrals. What you will learn in this Notebook1. to build a probabilistic model for a coin, dice, ...2. to sample from these models3. to answer queries either analytically or via samplingYou will also learn4. what a Bernoulli and Categoirical distribution are5. what a cumulative distribution is6. How to compute expectations (mean, variance)7. indpendence/dependence and conditional indpendence/dependence The following subsections summarize the coin model we discussed during the lecture and introduce the Dice model. Model of a coinVariables:* bias of the coin $\theta$ * outcome of the coin toss $x$Meaning of $\theta$: $\theta=0.5$ (fair coin), $\theta=1$ (two heads),  $\theta=0$ (two tails).We denote with* $x$ the outcome of the coin toss* $dom(x) = \{1,0\}$We use 1 for Heads and 0 for Tail. Probabilistic model$x$ is a binary variable and, its distribution can in general be written as:$$\left\{\begin{array}{rcl}p(x=1)&=&\theta\\p(x=0)&=&1-\theta\end{array}\right.$$or, in a line,$$p(x=\textsf{x})=\theta^{\textsf{x}}(1-\theta)^{1-\textsf{x}} ~~~~ \text{ for } ~~~~ \textsf{x}\in\{0,1\}$$This is called **Bernoulli distribution** ExamplesFor instance, for a fair coin ($\theta=0.5$),$$\left\{\begin{array}{ll}p(x=1)&=&0.5\\p(x=0)&=&0.5\end{array}\right.$$Here an example of a tricked coin$$\left\{\begin{array}{ll}p(x=1)&=&\frac{2}{3}\\p(x=0)&=&\frac{1}{3}\end{array}\right.$$ Dice* x denotes the result of the dice thrown* dom(x)=$\{1,2,3,4,5,6\}$The probability mass functions$$\begin{array}{l}p(x=1) = \theta_1\\ p(x=2) = \theta_2\\ p(x=3) = \theta_3\\ p(x=4) = \theta_4\\ p(x=5) = \theta_5\\ p(x=6) = \theta_6\\ \end{array}$$where $\theta_i\geq0$ is the probability that the outcome is the i-th face and, therefore, $\sum_{i=1}^6\theta_i=1$.Example Fair Dice:$$\begin{array}{l}p(x=1) = \frac{1}{6}\\ p(x=2) = \frac{1}{6}\\ p(x=3) = \frac{1}{6}\\ p(x=4) = \frac{1}{6}\\ p(x=5) = \frac{1}{6}\\ p(x=6) = \frac{1}{6}\\ \end{array}$$ Categorical distributionWe can encode the six possible outcomes as follows$$\begin{array}{rcl}1 &\rightarrow& [1,0,0,0,0,0]\\2 &\rightarrow& [0,1,0,0,0,0]\\3 &\rightarrow& [0,0,1,0,0,0]\\4 &\rightarrow& [0,0,0,1,0,0]\\5 &\rightarrow& [0,0,0,0,1,0]\\6 &\rightarrow& [0,0,0,0,0,1]\\\end{array}$$then if we call the generic binary vector as $y$ and its $i$-th component as $y_i$, we can write$$p(x=y)=\theta_1^{y_1}\theta_2^{y_2}\theta_3^{y_3}\theta_4^{y_4}\theta_5^{y_5}\theta_6^{y_6}$$example $p(x=[0,1,0,0,0,0])=\theta_2$.The above distribution is called **Categorical distribution** (we have seen it for the case of six categories (dice), but we can extend it to any generic number of categories). ExpectationGive a function $f(x)$ of the variable $x$, its expectation with respect the probability distributionof $x$ is equal$$E[f(x)]=\sum_{\mathscr{x}\in dom(x)} f(\mathscr{x})p(x=\mathscr{x})$$If* $f(x)=x$ (identity function) then $E[x]$ is the mean of $x$;* $f(x)=(x-E[x])^2$ then $E[(x-E[x])^2]=Var(x)$ is called the variance of $x$;* $f(x)=(x==a)$  then $E[(x==a)]$ is the probability of $x=a$;* $f(x)=(x>=a)$  then $E[(x>=a)]$ is the probability of $x\geq a$;Example Fair dice: $$\begin{aligned}E[x]&=1\frac{1}{6}+2\frac{1}{6}+3\frac{1}{6}+4\frac{1}{6}+5\frac{1}{6}+6\frac{1}{6}=3.5\\E[x^2]&=1\frac{1}{6}+4\frac{1}{6}+9\frac{1}{6}+16\frac{1}{6}+25\frac{1}{6}+36\frac{1}{6}=13.5\\Var(x)=E[(x-3.5)^2]&=(1-3.5)^2\frac{1}{6}+(2-3.5)^2\frac{1}{6}+(3-3.5)^2\frac{1}{6}+(4-3.5)^2\frac{1}{6}+(5-3.5)^2\frac{1}{6}+(6-3.5)^2\frac{1}{6}=\frac{35}{12}\\E[x==1]&=1\frac{1}{6}+0\frac{1}{6}+0\frac{1}{6}+0\frac{1}{6}+0\frac{1}{6}+0\frac{1}{6}=\frac{1}{6}\\E[x>=2]&=0\frac{1}{6}+1\frac{1}{6}+1\frac{1}{6}+1\frac{1}{6}+1\frac{1}{6}+1\frac{1}{6}=\frac{5}{6}\\\end{aligned}$$ StatisticsHow do we do statistical analysis?  How can we test if a coin is fair or not?Assume we observe the following outcome:$$Observations=[1,1,1,1,1]$$is the coin fair?  Bag of coins modelFor the moment, let us assume we have only three types of coins1. $\theta_1=0.25$2. $\theta_2=0.5$3. $\theta_3=0.75$and the number of coin of type 1,2 and 3 in the bag is the same.We can build a model that describes the way the observations could be generated.    A coin is extracted at random from a bag containing the coins of type 1, 2 and 3.    The coin is then tossed five times.  PosteriorWe want to compute the posterior probabilities1. $p(\theta_1=0.25 ~|~ [1,1,1,1,1])$2. $p(\theta_2=0.5~|~[1,1,1,1,1])$3. $p(\theta_3=0.75~|~[1,1,1,1,1])$and use these probabilities to answer the following queries:    *Is the coin fair?*Is the coin biased towards Head?<jupyter_code># SYMPY libray
from sympy.stats import Bernoulli, sample 

θ = 0.5    
C = Bernoulli("Coin", θ)
C
# sympy.stats library
# https://docs.sympy.org/latest/modules/stats.html
from sympy.stats import density

def pmf(D):
    return (list(density(D).dict.keys()),list(density(D).dict.values()))
print(pmf(C))
# matplotlib
# https://matplotlib.org/
import matplotlib.pyplot as plt
%matplotlib inline

plt.plot(pmf(C)[0], pmf(C)[1], 'o', ms=8)
plt.vlines(pmf(C)[0], 0, pmf(C)[1], linestyles='-', lw=2 )
print(pmf(C))
plt.plot(pmf(C)[0], pmf(C)[1], 'o', ms=8)
plt.vlines(pmf(C)[0], 0, pmf(C)[1], linestyles='-', lw=2 )<jupyter_output>([0, 1], [0.500000000000000, 0.500000000000000])
{'fraction_non_alphanumeric': 0.19532810819117874, 'fraction_numerical': 0.06008913477793146, 'mean_word_length': 5.312318137730359, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 8, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '96966', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2521, 'n_tokens_neox': 2378, 'n_words': 692, 'path': 'Weeks_1-2/Labs/CS6462_Lab_1_3.ipynb', 'repo_name': 'goksuturann/CS6462_SEM2_2021-2', 'size': '6515'}