<jupyter_start><jupyter_text>MLE and Hypothesis Testing  Read the given csv file [```q4.csv```](https://raw.githubusercontent.com/Ipsit1234/SHALA/master/q4.csv) of samples. Assume that it is drawn from a exponential distribution. What is the difference of log likelihoods for $\lambda = 1$ and $\lambda = 2$ ? Which of the two is more strongly supported?<jupyter_code>import numpy as np
import pandas as pd
from scipy import stats
from matplotlib import pyplot as plt
import seaborn as sns
url = 'https://raw.githubusercontent.com/Ipsit1234/SHALA/master/q4.csv'
data = pd.read_csv(url,header=None)
data.head()<jupyter_output><empty_output><jupyter_text>PDF of exponential distribution is $\lambda e^{-\lambda x}$<jupyter_code>def pdf_exp(lamb,x):
	return lamb * np.exp(-1*lamb*x)

likelihood1 = 1
likelihood2 = 1
for aa in data[1].values:
  likelihood1 = likelihood1 * pdf_exp(x = aa,lamb=1)
  likelihood2 = likelihood2 * pdf_exp(x = aa,lamb=2)
log_likelihood1 = np.log(likelihood1)
log_likelihood2 = np.log(likelihood2)
print(log_likelihood1 - log_likelihood2)<jupyter_output>-20.366416970604874<jupyter_text>Read the csv files [```q4_1.csv```](https://raw.githubusercontent.com/Ipsit1234/SHALA/master/q4_1.csv) and [```q4_2.csv```](https://raw.githubusercontent.com/Ipsit1234/SHALA/master/q4_2.csv) to get samples from two unknown distributions. Choose a significance level and implement the the most general two-sample t-test<jupyter_code>sample1 = pd.read_csv('https://raw.githubusercontent.com/Ipsit1234/SHALA/master/q4_1.csv',header=None)
sample2 = pd.read_csv('https://raw.githubusercontent.com/Ipsit1234/SHALA/master/q4_2.csv',header=None)
print(sample1.head())
print(sample2.head())<jupyter_output>0         1
0  0 -0.179278
1  1  0.236714
2  2  0.255502
3  3 -0.029321
4  4  0.004818
   0         1
0  0  0.680541
1  1  0.436810
2  2  0.604089
3  3  0.518702
4  4  0.479711<jupyter_text>Find the parameters required to compute the T statistic<jupyter_code>significance = 0.05
mu1 = np.mean(sample1[1].values)
mu2 = np.mean(sample2[1].values)
s1 = np.var(sample1[1].values)
s2 = np.var(sample2[1].values)
n1 = sample1[1].shape[0]
n2 = sample1[1].shape[0]<jupyter_output><empty_output><jupyter_text>Find the degrees of freedom to get the critical value<jupyter_code>T = (mu1 - mu2) / np.sqrt((s1/n1) + (s2 / n2))
df = np.square((s1/n1) + (s2 / n2)) / (((s1/n1)**2)/(n1-1) + ((s2/n2)**2)/(n2-1))<jupyter_output><empty_output><jupyter_text>Compare the critical value and the T statistic and conclude whether null hypothesis is accepted or not. Print 'YES' is it is rejected, otherwise print 'NO'.<jupyter_code>critical_value = stats.t.pdf((1-significance)/2,df)
print('T : {}, Critical Value : {}'.format(np.abs(T),critical_value))
if np.abs(T) > critical_value:
#   print('Null hypothesis is rejected')
    print('YES')
else:
#   print('Null hypothesis is not rejected')
    print('NO')<jupyter_output>T : 18.292145241166835, Critical Value : 0.35547474461325235
YES<jupyter_text>Plot the distribution of both the samples to visualize<jupyter_code>sns.distplot(sample1[1].values)

sns.distplot(sample2[1].values)
plt.legend(['first sample','second sample'])
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13884555382215288, 'fraction_numerical': 0.08049921996879875, 'mean_word_length': 5.335968379446641, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 27, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '417595', 'license': 'MIT', 'n_tokens_mistral': 1342, 'n_tokens_neox': 1219, 'n_words': 268, 'path': '04-Assignment/Solution/Assignment.ipynb', 'repo_name': 'hritik5102/SHALA2020', 'size': '3205'}