<jupyter_start><jupyter_text>*Data Science Unit 1 Sprint 3 Assignment 1* Apply the t-test to real dataYour assignment is to determine which issues have "statistically significant" differences between political parties in this [1980s congressional voting data](https://archive.ics.uci.edu/ml/datasets/Congressional+Voting+Records). The data consists of 435 instances (one for each congressperson), a class (democrat or republican), and 16 binary attributes (yes or no for voting for or against certain issues). Be aware - there are missing values!Your goals:1. Load and clean the data (or determine the best method to drop observations when running tests)2. Using hypothesis testing, find an issue that democrats support more than republicans with p < 0.013. Using hypothesis testing, find an issue that republicans support more than democrats with p < 0.014. Using hypothesis testing, find an issue where the difference between republicans and democrats has p > 0.1 (i.e. there may not be much of a difference)Note that this data will involve *2 sample* t-tests, because you're comparing averages across two groups (republicans and democrats) rather than a single group against a null hypothesis.Stretch goals:1. Refactor your code into functions so it's easy to rerun with arbitrary variables2. Apply hypothesis testing to your personal project data (for the purposes of this notebook you can type a summary of the hypothesis you formed and tested) +I'm adding to the assignment that when you have already done what's asked of you there, before you move onto the other stretch goals, that:5. You also practice some 1-sample t-tests6. You try and create some kind of a visualization that communicates the results of your hypothesis tests. This can be as simple as a histogram of the p-values or the t-statistics.  Part 1: Load & Clean The Data<jupyter_code>### YOUR CODE STARTS HERE
#importing libraries
import pandas as pd
import numpy as np 
from scipy.stats import ttest_ind, ttest_ind_from_stats, ttest_rel, t, ttest_1samp
import seaborn as sns
from matplotlib import style
import matplotlib.pyplot as plt

#loading file
!wget https://archive.ics.uci.edu/ml/machine-learning-databases/voting-records/house-votes-84.data
# checking to see what files are in the current main directory  
!ls
# Loading data, creating dataframe df with a custom header
df = pd.read_csv('house-votes-84.data', 
                 header=None,
                 names=['party','handicapped-infants','water-project',
                          'budget','physician-fee-freeze', 'el-salvador-aid',
                          'religious-groups','anti-satellite-ban',
                          'aid-to-contras','mx-missile','immigration',
                          'synfuels', 'education', 'right-to-sue','crime','duty-free',
                          'south-africa'])

#inspecting data
df.shape


#inspecting data
df.head(5)<jupyter_output><empty_output><jupyter_text>Here, you can see question marks, that is no good. We must a do something about a that. Next we need to replace the question marks with NaN values - AND set those NaN value back into a new recast DataFrame df. Also we will change the string yes, no (yea, nay) votes to binary 1,0 integers.<jupyter_code><jupyter_output><empty_output><jupyter_text><jupyter_code># cleaning: +NaN, string to int
df = df.replace({'?':np.NaN, 'n':0, 'y':1})


#inspecting
df.shape
#instecting
df.head(5)

# inspecting
# Looking over abstensions...
#
# "How long can the Britsh hang on in Gebralter, 
# where the tapestries of simitared riders hunt tigers... 
# clinging to the rocks like rock apes, 
# clinging always to less and less."
df.isnull().sum()
#use "filtering" to create two new party based df (so much for nonpartisan dataframes...a sad day)
dem = df[df['party'] == 'democrat']
rep = df[df['party'] == 'republican']
#inspect
dem.head(5)
#inspect
rep.head(5)<jupyter_output><empty_output><jupyter_text>2. Using hypothesis testing, find an issue that democrats support more than republicans with p < 0.01<jupyter_code>print(ttest_ind(rep['handicapped-infants'], dem['handicapped-infants'], nan_policy='omit'))
print(rep['handicapped-infants'].mean(), 'Republican mean')
print(dem['handicapped-infants'].mean(), 'Democratic mean')<jupyter_output>Ttest_indResult(statistic=-9.205264294809222, pvalue=1.613440327937243e-18)
0.18787878787878787 Republican mean
0.6046511627906976 Democratic mean<jupyter_text>Here the small pvalue below .5, indicates a lack of similarity between the two means compared. Here the samll p-value indicates a difference beyond chance between the two. The null hypothesis was th Ttest_indResult(statistic=-9.205264294809222, pvalue=1.613440327937243e-18)0.18787878787878787 Republican mean0.6046511627906976 Democratic mean<jupyter_code>handi = ttest_ind(rep['handicapped-infants'], dem['handicapped-infants'], nan_policy='omit')
#import matplotlib.pyplot as plt
y1 = rep['handicapped-infants'].dropna()
y2 = dem['handicapped-infants'].dropna()

fix, ax = plt.subplots()
for sample in [y1, y2]:
  sns.distplot(sample)<jupyter_output><empty_output><jupyter_text>A visual comparison of democratic and republican votes. 3. Using hypothesis testing, find an issue that republicans support more than democrats with p < 0.01<jupyter_code>print(ttest_ind(rep['religious-groups'], dem['religious-groups'], nan_policy='omit'))
print(rep['religious-groups'].mean(), 'Republican mean')
print(dem['religious-groups'].mean(), 'Democratic mean')<jupyter_output>Ttest_indResult(statistic=9.737575825219457, pvalue=2.3936722520597287e-20)
0.8975903614457831 Republican mean
0.47674418604651164 Democratic mean<jupyter_text>Due to our test resulting in a T-Statistic of 9.737575825219457 and a P-value of 2.3936722520597287e-20: we reject the null hypothesis that the two vote means are not different.Here the small pvalue below .5, indicates a lack of similarity between the two means compared.I think: The null hypothesis is that they are not different, and the result showing that they are different means strongly rejecting (the null hypothesis) that they are not different (a.k.a. this is evidence that they are are different (in a way beyond mere change result).Ttest_indResult(statistic=9.737575825219457, pvalue=2.3936722520597287e-20)0.8975903614457831 Republican mean0.47674418604651164 Democratic mean<jupyter_code>#import matplotlib.pyplot as plt
y1 = rep['religious-groups'].dropna()
y2 = dem['religious-groups'].dropna()

fix, ax = plt.subplots()
for sample in [y1, y2]:
  sns.distplot(sample)<jupyter_output><empty_output><jupyter_text>A visual comparison of democratic and republican votes. 4. Using hypothesis testing, find an issue where the difference between republicans and democrats has p > 0.1 (i.e. there may not be much of a difference)<jupyter_code>print(ttest_ind(rep['water-project'], dem['water-project'], nan_policy='omit'))
print(rep['water-project'].mean(), 'Republican mean')
print(dem['water-project'].mean(), 'Democratic mean')<jupyter_output>Ttest_indResult(statistic=0.08896538137868286, pvalue=0.9291556823993485)
0.5067567567567568 Republican mean
0.502092050209205 Democratic mean<jupyter_text>Here the pvalue above .5, indicates a similarity between the two means compared.I think: The null hypothesis is that they are not different, and the result showing that they are very similar falls in the catagory of failing to disprove (the null hypothesis) that they are different (a.k.a. this is evidence that they are similar).And the T value being near 0 shows they are similar...Ttest_indResult(statistic=0.08896538137868286, pvalue=0.9291556823993485)0.5067567567567568 Republican mean0.502092050209205 Democratic mean<jupyter_code>#import matplotlib.pyplot as plt
y1 = rep['water-project'].dropna()
y2 = dem['water-project'].dropna()

fix, ax = plt.subplots()
for sample in [y1, y2]:
  sns.distplot(sample)<jupyter_output><empty_output><jupyter_text>A visual comparison of democratic and republican votes. 5. Practice some 1-sample t-tests<jupyter_code>#single sample t-tests
# passing nan_policy='omit' 
ttest_1samp(rep['budget'], 0, nan_policy='omit')
#single sample t-tests
# passing nan_policy='omit' 
ttest_1samp(rep['budget'], 1, nan_policy='omit')
ttest_1samp(dem['budget'], 0, nan_policy='omit')
ttest_1samp(dem['budget'], 1, nan_policy='omit')

ttest_1samp(rep['water-project'], 0, nan_policy='omit')

ttest_1samp(dem['water-project'], 0, nan_policy='omit')<jupyter_output><empty_output><jupyter_text>6. You try and create some kind of a visualization that communicates the results of your hypothesis tests. This can be as simple as a histogram of the p-values or the t-statistics.  Ttest_indResult(statistic=9.737575825219457, pvalue=2.3936722520597287e-20)0.8975903614457831 Republican mean0.47674418604651164 Democratic meanTtest_indResult(statistic=0.08896538137868286, pvalue=0.9291556823993485)0.5067567567567568 Republican mean0.502092050209205 Democratic meanTtest_indResult(statistic=9.737575825219457, pvalue=2.3936722520597287e-20)0.8975903614457831 Republican mean0.47674418604651164 Democratic mean<jupyter_code><jupyter_output><empty_output><jupyter_text><jupyter_code>#https://pythonspot.com/matplotlib-bar-chart/

import matplotlib.pyplot as plt; plt.rcdefaults()

fig = plt.figure()

objects = ('Handicaped Kids(Dem Issue)', 'Water Bill', 'Religion(GOP issue)')
y_pos = np.arange(len(objects))
performance = [9.737575825219457,0.08896538137868286,9.737575825219457]


plt.bar(y_pos, performance, align='center', alpha=0.5)
plt.xticks(y_pos, objects)
plt.ylabel('Levels of Partisan Voting')
plt.title('Republicans & Democrats Agreed on Water in 1984')


plt.text(x=-.5, y=-1.5, s="Partisanship As reflected in T-Score of Average Party Votes", 
        fontweight='bold', fontsize=10);
plt.text(x=-.9, y=-1.9, s="Data Source: https://archive.ics.uci.edu/ml/machine-learning-databases/voting-records/house-votes-84.data", 
        fontweight='bold', fontsize=7);

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10184907546226886, 'fraction_numerical': 0.08095952023988005, 'mean_word_length': 5.561311475409836, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 48, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '125775', 'license': 'MIT', 'n_tokens_mistral': 3646, 'n_tokens_neox': 3152, 'n_words': 991, 'path': 'module1-statistics-probability-and-inference/GGA_131_v5_asnmt_LS_DS_131_Statistics_Probability_Assignment.ipynb', 'repo_name': 'lineality/DS-Unit-1-Sprint-3-Statistical-Tests-and-Experiments', 'size': '10005'}