<jupyter_start><jupyter_text>Distributions of Human Development Indices in the USABefore we progress to modeling it may be illustrative to examine the distribution of some of the indices we calculated.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline
df = pd.read_csv('./data/clean_data.csv')
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 2969 entries, 0 to 2968
Columns: 194 entries, Description to HDI
dtypes: float64(191), object(3)
memory usage: 4.4+ MB<jupyter_text>The chart below demonstrates that the HDIs are fairly normally distributed accross the counties in our dataset.  This is good to see because it suggests that the model may be able to extract some differentability.<jupyter_code>sns.color_palette("YlOrBr", as_cmap=True)
sns.violinplot(x = df['HDI'], color = "brown").set(title = "Density of HDI Values")
plt.savefig("./assets/hdi", transparent=True)<jupyter_output><empty_output><jupyter_text>We also see that the Income Index (II) is normally distributed.<jupyter_code>sns.color_palette("YlOrBr", as_cmap=True)
sns.violinplot(x = df['II'], color = "brown").set(title = "Density of II Values")
plt.savefig("./assets/ii", transparent=True)<jupyter_output><empty_output><jupyter_text>The Educational Index (EI) is also fairly normal, but note that it is a narrower distribution than income.  This suggests that years of education are fairly standardized in the United States with relatively few counties having a much larger proportion of highschool dropouts or advanced degree earners from the mean.<jupyter_code>sns.color_palette("YlOrBr", as_cmap=True)
sns.violinplot(x = df['EI'], color = "brown").set(title = "Density of EI Values")
plt.savefig("./assets/ei", transparent=True)<jupyter_output><empty_output><jupyter_text>We do not see the same pattern in the Life Expectancy Index (LE). It appears that there are a large number of counties that see considerably lower life expectancies than the American Mean.  We may expect to see a lot of impact on health and wellness features in our model of the HDI.<jupyter_code>sns.color_palette("YlOrBr", as_cmap=True)
sns.violinplot(x = df['LE'], color = "brown").set(title = "Density of LI Values")
plt.savefig("./assets/le", transparent=True)<jupyter_output><empty_output><jupyter_text>Finally, the large spread on the box plot below shows that there are many features that are highly correlated to the II.  This suggests that in the next step we will remove a lot of income related features to reduce colinearity in our features.<jupyter_code>sns.color_palette("YlOrBr", as_cmap=True)
sns.boxplot(x = df.corr()['II'], color = "brown").set(title = "Feature Correlation to II")
plt.savefig("./assets/iicorr", transparent=True)<jupyter_output><empty_output><jupyter_text>We see a similar pattern in the correlations to EI, although here the spread is somewhat smaller, so we should not expect to need to remove as many educational features.<jupyter_code>sns.color_palette("YlOrBr", as_cmap=True)
sns.boxplot(x = df.corr()['EI'], color = "brown").set(title = "Feature Correlation to EI")
plt.savefig("./assets/eicorr", transparent=True)<jupyter_output><empty_output><jupyter_text>Lastly, colinearity of features with respect to the LE is moderate.  Some of these features will likely be removed as well, but not to the extent of the incoome related features.<jupyter_code>sns.color_palette("YlOrBr", as_cmap=True)
sns.boxplot(x = df.corr()['LE'], color = "brown").set(title = "Feature Correlation to LE")
plt.savefig("./assets/lecorr", transparent=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10375034218450589, 'fraction_numerical': 0.005474952094169176, 'mean_word_length': 6.366935483870968, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '625118', 'license': 'MIT', 'n_tokens_mistral': 1183, 'n_tokens_neox': 1121, 'n_words': 391, 'path': 'Detroit_HDI/20 - eda.ipynb', 'repo_name': 'jwarc927/Portfolio', 'size': '3653'}