<jupyter_start><jupyter_text>EDA of the state.x77 R dataset  3rd discussion exercise for the Statistical-Thinking-for-Data-Science MOOC course in edX Importing python libraries.<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
sns.set()<jupyter_output><empty_output><jupyter_text>Importing data:<jupyter_code>df = pd.read_csv('/home/caio/Desktop/edX/ColumbiaX-Statistical-Thinking-for-Data-Science/statex77.csv')<jupyter_output><empty_output><jupyter_text>First five states in the dataset:<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>In this analysis I wanted to categorize one of the variables. It's funny how in the last two lessons, she taught us test of independence and ANOVA, and how both tests deal with at least one categorical variable and in the end she provide us a dataset with only quantitative variables. Hence, I chose a variable out of whim: HS Grad, the percentage of population with High school degrees in that state in 1970. By having a look at the histogram of that variable, it seemed to me that dividing the x axis in 3 regions would be a great way to discretize HS Grad. The ranges I chose were from 0 to 45, from 45 to 55 and from 55 to 100. The idea is to perform an ANOVA test with this newly created variable and some other later on. Have a look, and see if you agree with the way I discretized HS Grad:<jupyter_code>df['HS Grad'].hist(bins=20)
plt.ylabel('counts')
plt.xlabel('HS Grad')
plt.show()<jupyter_output><empty_output><jupyter_text>Here I actualy categorize the variable and labeled the ranges as low, mid and high:<jupyter_code>df['HSGrad_cat'] = pd.cut(df['HS Grad'], bins=[0,45,55,100], include_lowest=True, labels=['low','mid','high'])<jupyter_output><empty_output><jupyter_text>The categories are not equally populated, but that's ok.<jupyter_code>df['HSGrad_cat'].value_counts()<jupyter_output><empty_output><jupyter_text>Here I make a bar plot to assess the association between the HS Grad variable (categorized) and the Illiteracy variable. Naturally I would expect that the higher the number of high school graduates, the lower the illiteracy. That seems to be the case. But notice that there isn't much a difference between the high and mid categories. Maybe that has to do with the way I categorized HS Grad? Who knows. I leave it up to you to find out.<jupyter_code>sns.barplot(df['HSGrad_cat'],df['Illiteracy'])
plt.show()<jupyter_output><empty_output><jupyter_text>Here I tried to look at the actual distributions and not just the averages in each population. That gives us a more clear picture of what is going on.<jupyter_code>plt.hist(df[df['HSGrad_cat']=='low']['Illiteracy'],bins=7,range=(0.5,2.5),alpha=0.5,density=True)
plt.hist(df[df['HSGrad_cat']=='mid']['Illiteracy'],bins=7,range=(0.5,2.5),alpha=0.5,density=True)
plt.hist(df[df['HSGrad_cat']=='high']['Illiteracy'],bins=7,range=(0.5,2.5),alpha=0.5,density=True)
plt.show()<jupyter_output><empty_output><jupyter_text>Finally, I perform an ANOVA test to be certain of whether there is an association between the HS Grad groups and the Illiteracy variable. The statistic generated by the one way ANOVA is the F statistic, which represents the ratio of between group variability over the within group variability (Refer to: https://en.wikipedia.org/wiki/F-testMultiple-comparison_ANOVA_problems). Observe how this ratio is very high, and the associated p-value is magnitudes under the commoly accepted 0.05 significance. This only confirms what we already know, there IS an association between Illiteracy and the HS Grad categorical variable.<jupyter_code>fval,pval = stats.f_oneway(df[df['HSGrad_cat']=='low']['Illiteracy'],df[df['HSGrad_cat']=='mid']['Illiteracy'],df[df['HSGrad_cat']=='high']['Illiteracy'])
print("F statistic: {} || p-value: {}".format(round(fval,2),round(pval,7)))<jupyter_output>F statistic: 21.68 || p-value: 2e-07<jupyter_text>The possibilities of futher EDA (exploratory data analysis) are almost limitless. Of course, to be more practical you need to know what you're looking for. Below, I display a pair plot where the instances are grouped according to the HS Grad category. Other interesting insights can be extracted from this analysis. I also confirmed here another assumption: States with less Murder tend to have more high school grads! Have a look and see if you can find other interesting relations!<jupyter_code>sns.pairplot(df,hue= 'HSGrad_cat')
plt.show()<jupyter_output><empty_output><jupyter_text>Before going away I leave you with this funny finding. The second strongest correlation that we observe for the Illiteracy variable is with the Frost variable! Maybe that has to do with how northern states in the US had better education in the 70s? You tell me, I not from America :)<jupyter_code>corrs = df.corr()
np.abs(corrs['Illiteracy']).sort_values(ascending=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09513229650575641, 'fraction_numerical': 0.013532619672793376, 'mean_word_length': 6.369047619047619, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '186658', 'license': 'MIT', 'n_tokens_mistral': 1574, 'n_tokens_neox': 1460, 'n_words': 576, 'path': 'EDA_state.x77_Rdataset.ipynb', 'repo_name': 'CaioMar/DataScience', 'size': '4951'}