<jupyter_start><jupyter_text>Introduction to Probability and Statistics AssignmentIn this assignment, we will use the dataset of diabetes patients taken [from here](https://www4.stat.ncsu.edu/~boos/var.select/diabetes.html).<jupyter_code>import pandas as pd
import numpy as np

df = pd.read_csv("../../data/diabetes.tsv",sep='\t')
df.head()<jupyter_output><empty_output><jupyter_text>In this dataset, columns as the following:* Age and sex are self-explanatory* BMI is body mass index* BP is average blood pressure* S1 through S6 are different blood measurements* Y is the qualitative measure of disease progression over one yearLet's study this dataset using methods of probability and statistics. Task 1: Compute mean values and variance for all values<jupyter_code>df.mean()

df.var()<jupyter_output><empty_output><jupyter_text>Task 2: Plot boxplots for BMI, BP and Y depending on gender<jupyter_code>import matplotlib.pyplot as plt

df.boxplot(column='BMI',by='SEX')
plt.xticks(rotation='vertical')
plt.show()
df.boxplot(column='BP',by='SEX')
plt.xticks(rotation='vertical')
plt.show()
df.boxplot(column='Y',by='SEX')
plt.xticks(rotation='vertical')
plt.show()<jupyter_output><empty_output><jupyter_text>Task 3: What is the the distribution of Age, Sex, BMI and Y variables?<jupyter_code>df['AGE'].hist(bins=20)
plt.suptitle('Age distribution of diabetes patients')
plt.xlabel('Age')
plt.ylabel('Count')
plt.show()
df['SEX'].hist(bins=3)
plt.suptitle('Sex distribution of diabetes patients')
plt.xlabel('Sex')
plt.ylabel('Count')
plt.show()
df['BMI'].hist(bins=20)
plt.suptitle('BMI distribution of diabetes patients')
plt.xlabel('BMI')
plt.ylabel('Count')
plt.show()
df['Y'].hist(bins=30)
plt.suptitle('Y distribution of diabetes patients')
plt.xlabel('Y')
plt.ylabel('Count')
plt.show()<jupyter_output><empty_output><jupyter_text>Task 4: Test the correlation between different variables and disease progression (Y)> **Hint** Correlation matrix would give you the most useful information on which values are dependent.<jupyter_code>progression = df['Y']
bmi = df['BMI']

print(f"Correlation = {np.corrcoef(bmi,progression)[0,1]}")
progression = df['Y']
sex = df['SEX']

print(f"Correlation = {np.corrcoef(sex,progression)[0,1]}")
progression = df['Y']
bp = df['BP']

print(f"Correlation = {np.corrcoef(bp,progression)[0,1]}")<jupyter_output>Correlation = 0.4414817585625712<jupyter_text>Task 5: Test the hypothesis that the degree of diabetes progression is different between men and women<jupyter_code>df.groupby('SEX').agg({ 'AGE' : 'mean', 'BP' : 'mean', 'Y' : 'mean', 'SEX' : 'count'}).rename(columns={ 'SEX' : 'Count'})
# Student t-test
# Testataan, eteneekö diabetes nopeammin sukupuolella 1 kuin sukupuolella 2

# P-value can be considered as the probability of two distributions having the same mean. Koska tässä tapauksessa
# se on hyvin pieni, sukupuolten välillä vaikuttaa olevan eroa siinä miten tauti etenee.

# Negatiivinen T-arvo ilmeisesti viittaa siihen, että diabetes etenee nopeammin sukupuolella 2 kuin sukupuolella 1?

from scipy.stats import ttest_ind

tval, pval = ttest_ind(df.loc[df['SEX']== 1,['Y']], df.loc[df['SEX']== 2,['Y']],equal_var=False)
print(f"T-value = {tval[0]:.2f}\nP-value: {pval[0]}")<jupyter_output>T-value = -0.90
P-value: 0.3674449793083975
{'fraction_non_alphanumeric': 0.14442079659470963, 'fraction_numerical': 0.020370933414411676, 'mean_word_length': 6.927710843373494, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 24, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '34105', 'license': 'MIT', 'n_tokens_mistral': 1258, 'n_tokens_neox': 1157, 'n_words': 300, 'path': '1-Introduction/04-stats-and-probability/assignment.ipynb', 'repo_name': 'anmatela/Data-Science-For-Beginners', 'size': '3289'}