<jupyter_start><jupyter_text>Inference:It appears as though black male have far more complaints as opposed to other races and genders. Exploring a Research question (important for your upcoming group project):Why is this the case? Could this be because this sub-group tends to send more complaints about police officers or could this be because this sub-group is a marginalized group being targeted? Exploring Answers:It is not yet answerable using correlation alone, but this raises an intriging human-centered question based on the data and analytics available to us to further explore. This is the type of question you should ask yourselves and explore in your assignment projects.  Let's explore this sub-group a little further What if we looked at the typical age and the type of allegations made among this subgroup?<jupyter_code>tempdf1 = police_allegations[police_allegations.complainant_ethnicity == 'White']
whitedf_male = tempdf1[tempdf1.complainant_gender == 'Male']

whitedf_male.head()

tempdf = police_allegations[police_allegations.complainant_ethnicity == 'Black']
blackdf_male = tempdf[tempdf.complainant_gender == 'Male']

blackdf_male.head()
f, ax = plt.subplots(figsize=(7, 3))
sns.countplot(y="fado_type", data=blackdf_male, color="c")
f, ax = plt.subplots(figsize=(7, 3))
sns.countplot(y="outcome_description", data=blackdf_male, color="c")<jupyter_output><empty_output><jupyter_text>Can we see any kinds of correlation within this subgroup to help us further?  Exploring Possible Research Questions:Perhaps particular Precincts are more prejudice. Could we explore whether there is any correlation between the Police Precinct and the allegations at hand toward this sub-group?<jupyter_code>cor = blackdf_male['complainant_age_incident'].corr(blackdf_male['precinct'])
print(cor)<jupyter_output>-0.06356817282908908<jupyter_text>Exploring Possible InferencesAs we can see, there is a very weak negative correlation between the complainants age and the Police Precinct. Thus these variables do not have a strong correlation and inferences cannot be made with these variables. In your projects, you may arrive at such conclusions. To this point, we urge you to continue playing with different variables and seeing what open-ended human-centered research questions you can come up with. Your research questions and inferences are far more important than the answers because as data science students, the exploration of your research questions will drive you in this project.  Hypothesis Testing with Python<jupyter_code>import matplotlib
import pandas as pd
import matplotlib.pyplot as plt
import statsmodels
import numpy as np
from scipy import stats
import scipy.stats
import math
import random
import statistics
import scipy.stats as sp<jupyter_output><empty_output><jupyter_text>Now let's run a T-test<jupyter_code>print (sp.stats.ttest_ind(blackdf_male.dropna()['complainant_age_incident'], blackdf_male.dropna()['precinct']))
print (sp.stats.ttest_ind(blackdf_male.dropna()['complainant_age_incident'], whitedf_male.dropna()['precinct']))<jupyter_output>Ttest_indResult(statistic=-79.23073099788843, pvalue=0.0)<jupyter_text>(1) Starting with a 1-sample t test<jupyter_code>from scipy.stats import ttest_1samp
x =  blackdf_male.dropna()['complainant_age_incident']
tscore, pvalue = ttest_1samp(x, popmean=20)
print("t Statistic: ", tscore)  
print("P Value: ", pvalue)<jupyter_output>t Statistic:  119.91272774655172
P Value:  0.0<jupyter_text>(2) Next: a 2-sample t test<jupyter_code>group1 = blackdf_male.dropna()['complainant_age_incident']
group2 = whitedf_male.dropna()['complainant_age_incident']
#find variance for each group
print(np.var(group1), np.var(group2))
#perform two sample t-test with equal variances
sp.ttest_ind(a=group1, b=group2, equal_var=True)<jupyter_output><empty_output><jupyter_text>Next let's intepret our resultsThe two hypotheses for this particular two sample t-test are as follows:H0: µ1 = µ2 (the two population means are equal)HA: µ1 ≠µ2 (the two population means are not equal)Because the p-value of our test (4.25e-43) is greater than alpha = 0.05, we fail to reject the null hypothesis of the test. We do not have sufficient evidence to say that the mean complainant age between the two populations is different.  (3) Paired Sample t-testIn order for the paired sample t-test results to be trusted, the following assumptions need to be met:The dependent variable (DV) must be continuous which is measured on an interval or ratio scaleThe observations are independentThe DV should be approximately normally distributedThe paired sample t-test is robust to this violation. If there is a violation of normality, as long as it’s not in a major violation the test results can be considered valid<jupyter_code>sp.ttest_rel(group1[:20], group2[:20])<jupyter_output><empty_output><jupyter_text>Interpretation A paired sample t-test was used to analyze the age of the complainants between the populations of black and white males to test if there was statistical significance between the age and number of allegations made by each population group. In comparing the two population groups, there was not a statistically significant relationship (p=0.255)  (4) Welch's Sample t-testWelch’s t-test is a nonparametric univariate test that tests for a significant difference between the mean of two unrelated groups. It is an alternative to the independent t-test when there is a violation in the assumption of equality of variances.The hypothesis being tested is:Null hypothesis (H0): u1 = u2, which translates to the mean of sample 1 is equal to the mean of sample 2Alternative hypothesis (HA): u1 ≠ u2, which translates to the mean of sample 1 is not equal to the mean of sample 2If the p-value is less than what is tested at, most commonly 0.05, one can reject the null hypothesis.<jupyter_code>stats.ttest_ind(group1, group2, equal_var = False)<jupyter_output><empty_output><jupyter_text>InterpretationThe p-value is significant, therefore one can reject the null hypothesis in support of the alternative.  Let's try a new dataset! "total public complaints"<jupyter_code>public_complaints = pd.read_csv('./Total_Public_Complaints_ASR_PCF_TBL_001.csv', index_col=0)
public_complaints.head()
adf=public_complaints[['Subtype','Count_']]
adf.head()
adf.describe()

Serious_complaints = adf[adf['Subtype'] == 'Conduct - Serious']['Count_']
Other_law= adf[adf['Subtype'] == 'Better dealt in Other Law']['Count_']

Other_law<jupyter_output><empty_output><jupyter_text>1-sample t test<jupyter_code>tscore, pvalue = ttest_1samp(Serious_complaints, popmean=20)
print("t Statistic: ", tscore)  
print("P Value: ", pvalue)<jupyter_output>t Statistic:  1.537368600394069
P Value:  0.1848107961241521<jupyter_text>2-sample t test<jupyter_code>#find variance for each group
print(np.var(Serious_complaints), np.var(Other_law))
#perform two sample t-test with equal variances
sp.ttest_ind(a=Serious_complaints, b=Other_law, equal_var=True)<jupyter_output>225.88888888888889 236.47222222222226<jupyter_text>Paired sample t test<jupyter_code>sp.ttest_rel(Serious_complaints, Other_law)<jupyter_output><empty_output><jupyter_text>(4) Welch's Sample t-test<jupyter_code>stats.ttest_ind(Serious_complaints, Other_law, equal_var = False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07712328767123287, 'fraction_numerical': 0.0263013698630137, 'mean_word_length': 6.734110169491525, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 48, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '635172', 'license': 'MIT', 'n_tokens_mistral': 2351, 'n_tokens_neox': 2166, 'n_words': 792, 'path': 'tutorials/hypothesis_testing-allegations and public complaints.ipynb', 'repo_name': 'awaudrey/inf2178h-experimentaldesign-hcds', 'size': '7300'}