<jupyter_start><jupyter_text>Mutual Information**Mutual information:** For two discrete random variables $X$ and $Y,$ the mutual information between $X$ and $Y,$ denoted as $I(X;Y),$ measures how many much information they share. Specifically,$$I(X;Y)\triangleq D(p_{X,Y}\parallel p_{X}p_{Y}),$$ where $p_{X}p_{Y}$ is the distribution we get if $X$ and $Y$ were actually independent (i.e., if $X$ and $Y$ were actually independent, then we know that the joint probability table would satisfy $\mathbb {P}(X=x,Y=y)=p_{X}(x)p_{Y}(y)).$The mutual information could be thought of as how far $X$ and $Y$ are from being independent, since if indeed they were independent, then $I(X;Y)=0.$On the opposite extreme, consider when $X=Y.$ Then we would expect $X$ and $Y$ to share the most possible amount of information. In this scenario, we can write $p_{X,Y}(x,y)=p_{X}(x)\mathbf{1}\{ x=y\},$ and so$$\begin{eqnarray}I(X;Y)&=&D(p_{X,Y}\parallel p_{X}p_{Y})\\&=&\sum_{x}\sum_{y}p_{X,Y}(x,y)\log_{2}\frac{1}{p_{X}(x)p_{Y}(y)}\\&&\quad  -\sum_{x}\sum_{y}p_{X,Y}(x,y)\log_{2}\frac{1}{p_{X,Y}(x,y)}\\&=&\sum_{x}\sum_{y}p_{X}(x)\mathbf{1}\{x=y\}\log_{2}\frac{1}{p_{X}(x)p_{Y}(y)}\\&&\quad  -\sum_{x}\sum_{y}p_{X}(x)\mathbf{1}\{x=y\}\log_{2}\frac{1}{p_{X}(x)\mathbf{1}\{x=y\}}\\&=&\sum_{x}p_{X}(x)\log_{2}\Big(\frac{1}{p_{X}(x)}\Big)^{2}-\sum_{x}p_{X}(x)\log_{2}\frac{1}{p_{X}(x)}\\&=&2\sum_{x}p_{X}(x)\log_{2}\frac{1}{p_{X}(x)}-\sum_{x}p_{X}(x)\log_{2}\frac{1}{p_{X}(x)}\\&=&\sum_{x}p_{X}(x)\log_{2}\frac{1}{p_{X}(x)}\\&=&H(X).\end{eqnarray}$$This is not surprising: if $X$ and $Y$ are the same, then the number of bits they share is exactly the average number of bits needed to store $X$ (or $Y$), namely $H(X)$ bits.  Exercise: Mutual InformationConsider the following joint probability table for random variables $X$ and $Y.$ We'll compute the mutual information $I(X; Y)$ of random variables $X$ and $Y$ step-by-step.| $X$ vs $Y$ | 0       | 1      | 2      ||:----------:|:-------:|:------:|:------:|| 0          | $0.10$  | $0.09$ | $0.11$ || 1          | $0.08$  | $0.07$ | $0.07$ || 2          | $0.18$  | $0.13$ | $0.17$ |Mutual information is about comparing the joint distribution of $X$ and $Y$ with what the joint distribution would be if $X$ and $Y$ were actually independent.In Python (where we won't explicitly store the labels of the rows and columns):```pythonimport numpy as npjoint_prob_XY = np.array([[0.10, 0.09, 0.11],                           [0.08, 0.07, 0.07],                           [0.18, 0.13, 0.17]])```The marginal distributions $p_X$ and $p_Y$ are given by:```pythonprob_X = joint_prob_XY.sum(axis=1)prob_Y = joint_prob_XY.sum(axis=0)```Next, we produce what the joint probability table would be if $X$ and $Y$ were actually independent:```pythonjoint_prob_XY_indep = np.outer(prob_X, prob_Y)```At this point, we have the joint distribution of $X$ and $Y$ (denoted $p_{X,Y}$) stored in code as `joint_prob_XY`, and also what the joint distribution would be if $X$ and $Y$ were independent (denoted $p_ X p_ Y$) stored in code as `joint_prob_XY_indep`. The mutual information of $X$ and $Y$ is precisely given by the KL divergence between $p_{X,Y}$ and $p_ X p_ Y:$$$I(X;Y) = D(p_{X,Y}\parallel p_{X}p_{Y}) = \sum _ x \sum _ y p_{X, Y}(x, y) \log _2 \frac{p_{X, Y}(x, y)}{p_ X(x) p_ Y(y)}.$$ **Question:** What is $I(X; Y)?$ Provide just the number and don't write “bits" at the end. We suggest that you code a Python function that computes the information divergence between any two distributions, and then you can just plug in `joint_prob_XY` and `joint_prob_XY_indep`.(Please be precise with at least 5 decimal places, unless of course the answer doesn't need that many decimal places. You could also put a fraction.)**Answer:** {{ans}}<jupyter_code>import numpy as np
joint_prob_XY = np.array([[0.10, 0.09, 0.11], 
                          [0.08, 0.07, 0.07], 
                          [0.18, 0.13, 0.17]])
prob_X = joint_prob_XY.sum(axis=1)
prob_Y = joint_prob_XY.sum(axis=0)
joint_prob_XY_indep = np.outer(prob_X, prob_Y)

f = lambda x, y: joint_prob_XY[x][y] * \
    np.log2(joint_prob_XY[x][y] / (prob_X[x] * prob_Y[y]))
ans = sum([f(x,y) for x in range(3) for y in range(3)])
ans = "{0:.5f}".format(ans)<jupyter_output><empty_output><jupyter_text>**Question:** Are X and Y independent?**Ans:** No.<jupyter_code>from numpy.linalg import norm
norm(joint_prob_XY - joint_prob_XY_indep, np.inf)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.19968553459119498, 'fraction_numerical': 0.027628032345013476, 'mean_word_length': 4.251179245283019, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '658262', 'license': 'MIT', 'n_tokens_mistral': 1729, 'n_tokens_neox': 1636, 'n_words': 482, 'path': 'week04/04 Mutual Information.ipynb', 'repo_name': 'infimath/Computational-Probability-and-Inference', 'size': '4452'}