<jupyter_start><jupyter_text>*Data Science Unit 1 Sprint 3 Assignment 1* Apply the t-test to real dataYour assignment is to determine which issues have "statistically significant" differences between political parties in this [1980s congressional voting data](https://archive.ics.uci.edu/ml/datasets/Congressional+Voting+Records). The data consists of 435 instances (one for each congressperson), a class (democrat or republican), and 16 binary attributes (yes or no for voting for or against certain issues). Be aware - there are missing values!Your goals:1. Load and clean the data (or determine the best method to drop observations when running tests)2. Using hypothesis testing, find an issue that democrats support more than republicans with p < 0.013. Using hypothesis testing, find an issue that republicans support more than democrats with p < 0.014. Using hypothesis testing, find an issue where the difference between republicans and democrats has p > 0.1 (i.e. there may not be much of a difference)Note that this data will involve *2 sample* t-tests, because you're comparing averages across two groups (republicans and democrats) rather than a single group against a null hypothesis.Stretch goals:1. Refactor your code into functions so it's easy to rerun with arbitrary variables2. Apply hypothesis testing to your personal project data (for the purposes of this notebook you can type a summary of the hypothesis you formed and tested)<jupyter_code>cv_data_url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/voting-records/house-votes-84.data'
import pandas as pd
cv_data = pd.read_csv(cv_data_url)
import numpy as np
cv_data.head()
cv_data.count()
# Let's clean the data now
cv_data = pd.read_csv(cv_data_url, header=None)
cv_data.head()
cv_data.shape
column_headers = ['Class Name', 'handicapped-infants', 
                  'water-project-cost-sharing', 
                  'adoption-of-the-budget-resolution', 'physician-fee-freeze', 
                  'el-salvador-aid', 'religious-groups-in-schools', 
                  'anti-satellite-test-ban', 'aid-to-nicaraguan-contras', 
                  'mx-missile', 'immigration', 'synfuels-corporation-cutback',
                  'education-spending', 'superfund-right-to-sue', 'crime',
                  'duty-free-exports', 'export-administration-act-south-africa']

df = pd.read_csv(cv_data_url, names=column_headers)
print(df.shape)
df.head()
df['Class Name'].unique()
df.set_index('Class Name')
df.shape
df = df.replace(to_replace='?', value=np.NaN)
df.isnull().sum()
df['adoption-of-the-budget-resolution'].value_counts()
df = df.replace(('y', 'n'), (1, 0)) # Tried True False
df_rep = df[df['Class Name'] == 'republican']
df_dem = df[df['Class Name'] == 'democrat']
df_rep.shape, df_dem.shape
# Now we have our clean and ready to use data
df_rep = df_rep.drop(columns='Class Name')
df_dem = df_dem.drop(columns='Class Name')
df_rep.shape, df_dem.shape
# Using hypothesis testing, find an issue that democrats support more than republicans with p < 0.01
df_dem.describe().T
df_rep.describe().T
# handicapped-infants democrats support more than republicans 
# Now I need to prove it with p < 0.01
# From mean value we can democrats has more Yes
df_dem['handicapped-infants'].value_counts()
df_rep['handicapped-infants'].value_counts()

df_dem_mean = df_dem['handicapped-infants'].mean()
df_dem_mean
df_rep_mean = df_rep['handicapped-infants'].mean()
df_rep_mean
from scipy.stats import ttest_ind, ttest_ind_from_stats, ttest_rel
ttest_ind(df_dem['handicapped-infants'], df_rep['handicapped-infants'], nan_policy='omit')
print(f"Standard Deviation: {df_republican.std()}")
print(f"Mean: {df_republican.mean()}")
print(f"Standard Deviation: {df_democrat.std()}")
print(f"Variance: {df_democrat.var()}")

# Using hypothesis testing, find an issue that republicans support more than democrats with p < 0.01<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11037301992846194, 'fraction_numerical': 0.010219724067450179, 'mean_word_length': 4.817236255572065, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 6, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '164167', 'license': 'MIT', 'n_tokens_mistral': 1256, 'n_tokens_neox': 1188, 'n_words': 383, 'path': 'module1-statistics-probability-and-inference/Evgenii_Dudeiko_DSPT3_131_Statistics_Probability_Assignment.ipynb', 'repo_name': 'Edudeiko/DS-Unit-1-Sprint-3-Statistical-Tests-and-Experiments', 'size': '3914'}