<jupyter_start><jupyter_text>Assignment 2 - Elementary Probability and Information Theory  Boise State University DL - Dr. Kennington Instructions and Hints:* This notebook loads some data into a `pandas` dataframe, then does a small amount of preprocessing. Make sure your data can load by stepping through all of the cells up until question 1. * Most of the questions require you to write some code. In many cases, you will write some kind of probability function like we did in class using the data. * Some of the questions only require you to write answers, so be sure to change the cell type to markdown or raw text* Don't worry about normalizing the text this time (e.g., lowercase, etc.). Just focus on probabilies. * Most questions can be answered in a single cell, but you can make as many additional cells as you need. * When complete, please export as HTML. Follow the instructions on the corresponding assignment Trello card for submitting your assignment.<jupyter_code>import pandas as pd 

data = pd.read_csv('data/pnp-train.txt',delimiter='\t',encoding='latin-1', # utf8 encoding didn't work for this
                  names=['type','name']) # supply the column names for the dataframe

# this next line creates a new column with the lower-cased first word
data['first_word'] = data['name'].map(lambda x: x.lower().split()[0])
data[:10]
data.describe()<jupyter_output><empty_output><jupyter_text>1. Write a probability function/distribution $P(T)$ over the types. Hints:* The Counter library might be useful: `from collections import Counter`* Write a function `def P(T='')` that returns the probability of the specific value for T* You can access the types from the dataframe by calling `data['type']`<jupyter_code>from collections import Counter
data_types = data['type'].values
length_type = len(data_types)
stats_type = Counter(data_types)
def P(T=''):
    return stats_type[T] / length_type<jupyter_output><empty_output><jupyter_text>2. What is `P(T='movie')` ?<jupyter_code>P('movie')<jupyter_output><empty_output><jupyter_text>3. Show that your probability distribution sums to one.<jupyter_code>import numpy as np

np.sum([P(x) for x in stats_type.keys()])<jupyter_output><empty_output><jupyter_text>4. Write a joint distribution using the type and the first word of the nameHints:* The function is $P2(T,W_1)$* You will need to count up types AND the first words, for example: ('person','bill)* Using the zip function was useful for me here<jupyter_code>W1 = data['first_word'].values
type_firstWord = list(zip(data_types, W1))
length_joint_dist = len(type_firstWord)
stats_joint_dist = Counter(type_firstWord)
def P2(T='', W1=''):
    return stats_joint_dist[(T, W1)] / length_joint_dist<jupyter_output><empty_output><jupyter_text>5. What is P2(T='person', W1='bill')? What about P2(T='movie',W1='the')?<jupyter_code>P2('person', 'bill')
P2('movie', 'the')<jupyter_output><empty_output><jupyter_text>6. Show that your probability distribution P(T,W1) sums to one.<jupyter_code>np.sum([P2(type_word[0], type_word[1]) for type_word in stats_joint_dist.keys()])<jupyter_output><empty_output><jupyter_text>7. Make a new function Q(T) from marginalizing over P(T,W1) and make sure that Q(T) sums to one.Hints:* Your Q function will call P(T,W1)* Your check for the sum to one should be the same answer as Question 3, only it calls Q instead of P.<jupyter_code>def Q(T='movie'):
    return np.sum([P2(type_word[0], type_word[1]) for type_word in stats_joint_dist.keys() if type_word[0] == T])
Q('movie')
np.sum([Q(types) for types in stats_type.keys()])<jupyter_output><empty_output><jupyter_text>8. What is the KL Divergence of your Q function and your P function for Question 1?* Even if you know the answer, you still need to write code that computes it.<jupyter_code>np.sum([-P(types) * np.log10(Q(types) / P(types)) for types in stats_type.keys()])<jupyter_output><empty_output><jupyter_text>9. Convert from P(T,W1) to P(W1|T) Hints:* Just write a comment cell, no code this time. * Note that $P(T,W1) = P(W1,T)$  $P(W_1|T) = \frac{P(W_1,T)}{P(T)}$  10. Write a function `Pwt` (that calls the functions you already have) to compute $P(W_1|T)$.* This will be something like the multiplication rule, but you may need to change something<jupyter_code>def Pwt(W1='', T=''):
    return P2(T, W1)/P(T)<jupyter_output><empty_output><jupyter_text>11. What is P(W1='the'|T='movie')?<jupyter_code>Pwt('the', 'movie')<jupyter_output><empty_output><jupyter_text>12. Use Baye's rule to convert from P(W1|T) to P(T|W1). Write a function Ptw to reflect this. Hints:* Call your other functions.* You may need to write a function for P(W1) and you may need a new counter for `data['first_word']`<jupyter_code>data_W1 = data['first_word'].values
length_W1 = len(data_W1)
stats_W1 = Counter(data_W1)

def Pw(W1=''):
    return stats_W1[W1] / length_W1
    
def Ptw(W1='', T=''):
    return Pwt(W1, T) * P(T) / Pw(W1)<jupyter_output><empty_output><jupyter_text>13  What is P(T='movie'|W1='the')?  What about P(T='person'|W1='the')? What about P(T='drug'|W1='the')? What about P(T='place'|W1='the') What about P(T='company'|W1='the')<jupyter_code>Ptw('the', 'movie')
Ptw('the', 'person')
Ptw('the', 'drug')
Ptw('the', 'place')
Ptw('the', 'company')<jupyter_output><empty_output><jupyter_text>14 Given this, if the word 'the' is found in a name, what is the most likely type? **Movie**  15. Is Ptw(T='movie'|W1='the') the same as Pwt(W1='the'|T='movie') the same? Why or why not?<jupyter_code>Ptw('the', 'movie')
Pwt('the', 'movie')<jupyter_output><empty_output><jupyter_text>No, because conditional distributions aren't communative.  16. Do you think modeling Ptw(T|W1) would be better with a continuous function like a Gaussian? Why or why not?* Not really beacuse words are countable, disecrete things. But there are ways....  17. Determine the right probability distribution* Load the `rivers.csv` file and print out a histogram* Which disitrbutions do you think would fit the data for the `x` column? Here is a list: https://en.wikipedia.org/wiki/Category:Continuous_distributions* Hint: look at some of the exponential distributions, and maybe the Pareto* Use the principle of Maximum Entropy to detemrine which of the two distributions would best fit your data (you can sum over all values in your data)* You will need to estimate the parameters needed for each distribution (you may need to write maximum likelihood estimation functions for estimating your parameters)* You can use built-in python functions to model the distribution, if they exist. Otherwise, you may need to write your own* Show the Maximum Entropy calculations. * Calculate the KL divergence on the two distributions* make nice markdown commments so I know where everything is<jupyter_code>%matplotlib inline  

import matplotlib.pyplot as plt
data = pd.read_csv('data/rivers.csv') 
data.describe()
data.x.hist()<jupyter_output><empty_output><jupyter_text>Get the data of the column "x"<jupyter_code>X = data['x']<jupyter_output><empty_output><jupyter_text>Based on the histogram plot, the Exponential distribution seems good fitting into the data. And we also choose Gaussian distribution for comparison (the one that is not very close to this histogram).In order to estimate the parameters of these two distribution, we use the maximum likelihood estimation (MLE).- For the Gaussian distribution, $N \sim (\mu, \sigma^2)$, we need to estimate the mean and standard deviation. And the MLE solution of the Gaussian distribution is: $\hat{\mu} = \frac{1}{N}\sum_{n=1}^{N}{x_n}$; $\hat{\sigma}^2 = \frac{1}{N}\sum_{n=1}^{N}{(x_n - \hat{\mu})^2}$- For the Exponential distribution, $E \sim (\lambda)$, we need to estimate the parameter $\lambda$, which is estimated by MLE as: $\hat{\lambda} = \frac{N}{\sum_{n=1}^{N}{x_n}}$. This is the same as the reciprocal of the sample mean.Compute the estimation of parameters of the selected distributions.<jupyter_code>X_mean = np.mean(X)
X_std = np.std(X)
X_lambda = 1 / X_mean<jupyter_output><empty_output><jupyter_text>After getting these parameters, we want to compare the Entropy of fitted distributions.Using `scipy.stats` distribution functions to compute the values of the probability density functions.<jupyter_code>from scipy.stats import norm
from scipy.stats import expon
norm_pdf = norm.pdf(X, X_mean, X_std)
expon_pdf = expon.pdf(X, loc=0, scale = 1 / X_lambda)<jupyter_output><empty_output><jupyter_text>Define a function to compute entropy:$$H(p) = -\sum_{n=1}^{N}{P(X_n)log(P(X_n))}$$<jupyter_code>def entropy(pdf):
    return -np.sum([p * np.log(p) for p in pdf])<jupyter_output><empty_output><jupyter_text>Entropy of the Gassian distribution<jupyter_code>entropy(norm_pdf)<jupyter_output><empty_output><jupyter_text>Entropy of the Exponential distribution<jupyter_code>entropy(expon_pdf)<jupyter_output><empty_output><jupyter_text>From the values of entropy, we can see that the exponential distribution has a better fit.Calculate the KL divergence$$D_{KL}(P || Q) = \sum_{n=1}^{N}{P(X_n)log(\frac{P(X_n)}{Q(X_n)})}$$<jupyter_code>np.sum(-np.multiply(norm_pdf, np.log(np.divide(norm_pdf, expon_pdf))))<jupyter_output><empty_output><jupyter_text>We can see that the value is close to 0, which means these two fitted distribution are close to each other.We can plot it.- sort the values of X for plot.<jupyter_code>sortedIndex = np.argsort(X)
data.x.plot(kind='hist', normed=True)

plt.plot(X[sortedIndex], norm_pdf[sortedIndex])
plt.plot(X[sortedIndex], expon_pdf[sortedIndex])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12164710813076278, 'fraction_numerical': 0.011211232187761945, 'mean_word_length': 6.086117297698589, 'pattern_counts': {'":': 0, '<': 93, '<?xml version=': 0, '>': 93, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '559911', 'license': 'MIT', 'n_tokens_mistral': 3210, 'n_tokens_neox': 3104, 'n_words': 1065, 'path': 'introds/classes/A1/prob_inf_theory_assignment_tian_key.ipynb', 'repo_name': 'gcaracas/bsu', 'size': '9544'}