<jupyter_start><jupyter_text>Inspecting Risk Factors--- My initial thoughts are that certain risk factors such as "Obesity", "Smoker", "Pneumonia", etc. may play a role in whether or not a COVID patient dies. I will look at the death rate by risk factor.<jupyter_code>#these files are required in order to do value mapping for the visualizations
with open('Data/post_cleaning_data_mapping.json') as json_file:
    mappings = json.load(json_file)<jupyter_output><empty_output><jupyter_text>Figure 1 in Report<jupyter_code>#plotting all the binary yes or no risk factors death rates in the same figure as well as Sex and Entity Location
fig,ax = plt.subplots(7,2)
fig.set_size_inches(8, 25)
ax[0,0] = utils.getDeathRates3(covid_data,'Pneumonia','Died',ax[0,0],mappings['Pneumonia'])
ax[0,1] = utils.getDeathRates3(covid_data,'Kidney Failure','Died',ax[0,1],mappings['Kidney Failure'])
ax[1,0] = utils.getDeathRates3(covid_data,'Hypertension','Died',ax[1,0],mappings['Hypertension'])
ax[1,1] = utils.getDeathRates3(covid_data,'COPD Diagnosis','Died',ax[1,1],mappings['COPD Diagnosis'])
ax[2,0] = utils.getDeathRates3(covid_data,'Diabetes','Died',ax[2,0],mappings['Diabetes'])
ax[2,1] = utils.getDeathRates3(covid_data,'Cardiovascular Disease','Died',ax[2,1],mappings['Cardiovascular Disease'])
ax[3,0] = utils.getDeathRates3(covid_data,'Immunosuppression','Died',ax[3,0],mappings['Immunosuppression'])
ax[3,1] = utils.getDeathRates3(covid_data,'Other Diseases','Died',ax[3,1],mappings['Other Diseases'])
ax[4,0] = utils.getDeathRates3(covid_data,'Obesity','Died',ax[4,0],mappings['Obesity'])
ax[4,1] = utils.getDeathRates3(covid_data,'Pregnant','Died',ax[4,1],mappings['Pregnant'])
ax[5,0] = utils.getDeathRates3(covid_data,'Asthma','Died',ax[5,0],mappings['Asthma'])
ax[5,1] = utils.getDeathRates3(covid_data,'Smoker','Died',ax[5,1],mappings['Smoker'])
ax[6,0] = utils.getDeathRates3(covid_data,'Sex','Died',ax[6,0],mappings['Sex'])
ax[6,1] = utils.getDeathRates3(covid_data,'Entity of Residence Grouped','Died',ax[6,1],mappings['Entity of Residence Grouped'])
fig.tight_layout()
#fig.savefig('Binary Variable Death Rates.png')<jupyter_output><empty_output><jupyter_text>Part of Figure  in Report<jupyter_code>#plotting the death rates by age group (every 10 years)
gap = 5
death_rates = []
x_labs = []
x_labeling = []
for i in range(0,120,gap):
    subset = covid_data[covid_data['Age'].between(i,i+gap)]
    death_rates.append(sum(subset['Died'])/len(subset['Died']))
    x_labs.append((i+i+gap)/2)
    x_labeling.append(str(i)+"-"+str(i+gap))

fig,ax = plt.subplots(1)    
fig.set_size_inches(10, 6)
ax.bar(x_labs, death_rates, align='center', alpha=0.5,width=gap*3/4)
ax.set_xlabel('Age Groups')
ax.set_xticks(x_labs)
plt.xticks(rotation=45)
ax.set_xticklabels(x_labeling)
ax.set_ylabel('Death Rate')
ax.set_title('Death Rate by Age Group')
fig.tight_layout()
#fig.savefig('Age Rates.png')<jupyter_output><empty_output><jupyter_text>Creating new Age FeatureThe pattern of death rate for different age groups doesn't follow closely to any pattern that I can think of. It certainly isn't linear from start to end. However, it does appear linear between the ages of 35 and 90. Before age 35, the death rate is mostly uniform with the exception of the 0-5 age group. After age 90, the death rate actually declines. I have a bias in thinking that being extra old doesn't give you an advantage when fighting COVID. My thoughts are that there are less observations in these age groups, and that this has not allowed their age groups death rates to become closer to their real estimated death rates. Therefore, I want to create a new age group that classifies all people under the age of 35 as being 35, since it appears as if their death rates are very close anyway. Then, for anybody over the age of 90, I want to classify them as being 90, since even though the death rates start to fall, I am biased in thinking they actually flat line after age 90.<jupyter_code>#creating new age feature
upper_limit = 90
lower_limit = 30
new_ages = []
for i in covid_data['Age']:
    if i < lower_limit:
        new_ages.append(lower_limit)
    elif i > upper_limit:
        new_ages.append(upper_limit)
    else:
        new_ages.append(i)
covid_data['New Age'] = new_ages
#plotting the death rates by the new age group (every 5 years)
gap = 5
death_rates = []
x_labs = []
x_labeling = []
for i in range(35,90,gap):
    subset = covid_data[covid_data['New Age'].between(i,i+gap)]
    death_rates.append(sum(subset['Died'])/len(subset['Died']))
    x_labs.append((i+i+gap)/2)
    x_labeling.append(str(i)+"-"+str(i+gap))

fig,ax = plt.subplots(1)    
fig.set_size_inches(10, 6)
ax.bar(x_labs, death_rates, align='center', alpha=0.5,width=gap*3/4)
ax.set_xlabel('New Age Groups')
ax.set_xticks(x_labs)
ax.set_xticklabels(x_labeling)
ax.set_ylabel('Death Rate')
ax.set_title('Death Rate by New Age Group')
fig.tight_layout()
#fig.savefig('New Age Rates.png')<jupyter_output><empty_output><jupyter_text>This looks very linear now. I think that what I did is potentially putting bias into the data, but I think it should help the models later on perform much better since the ages will now match more to a distribution.  Creating new Feature that sums number of conditions someone has<jupyter_code>#creating list of factors to include in the sum and summing them
covid_data['Total Factors'] = covid_data[['Pneumonia','Diabetes','COPD Diagnosis','Asthma','Immunosuppression',
                                         'Hypertension','Other Diseases','Cardiovascular Disease','Obesity','Kidney Failure']].sum(axis=1)
#plotting the death rates by total factors (every 1 factor)
gap = 1
death_rates = []
x_labs = []
x_labeling = []
for i in np.arange(0,11,gap):
    subset = covid_data[covid_data['Total Factors'].between(i,i+gap)]
    death_rates.append(sum(subset['Died'])/max(1,len(subset['Died'])))
    x_labs.append((i+i+gap)/2)
    x_labeling.append(str(i)+"-"+str(i+gap))

fig,ax = plt.subplots(1)    
fig.set_size_inches(10, 6)
ax.bar(range(1,12), death_rates, align='center', alpha=0.5,width=gap*3/4)
ax.set_xlabel('Number of Risk Factors')
ax.set_xticks(range(1,12))
ax.set_xticklabels(range(1,12))
ax.set_ylabel('Death Rate')
ax.set_title('Death Rate by Number of Factors Present')
#fig.savefig('Total Factors Rates.png')<jupyter_output><empty_output><jupyter_text>Before Exploring Interactions, I'd like to update the data file to have these two new features<jupyter_code>covid_data.to_csv('Data/Post_Visualization_10-16.csv',index=False)<jupyter_output><empty_output><jupyter_text>Exploring Interactions<jupyter_code># First I need to encode the features
enc = OneHotEncoder()
enc.fit(covid_data[['Entity of Residence Grouped','Sex']])
encoded = enc.transform(covid_data[['Entity of Residence Grouped','Sex']]).toarray()
covid_data[list(enc.get_feature_names(['Entity Group','Sex']))] = encoded
del covid_data['Sex']
del covid_data['Entity of Residence Grouped']
#I am removing these features from the list to create interactions among. The reason is that Died is target variable, Age has
#been replaced by New Age, but I dont' want to get rid of the original Age in the data, and I don't want to look at interactions
#with location based features
cols = list(covid_data.columns)
cols.remove('Died')
cols.remove('Age')
cols.remove('Entity Group_1')
cols.remove('Entity Group_2')
cols.remove('Entity Group_3')
#I'm creating the interaction terms and then writing them to a dataframe and then finding out each of their 
#correlations with the target variable 'Died' so I can get an idea of which ones to test in the modeling file
poly_features = PolynomialFeatures(degree=2, interaction_only=True)
poly_features.fit(covid_data[cols])
interactions_data = poly_features.transform(covid_data[cols])
interaction_names = poly_features.get_feature_names(cols)
interactions_data = pd.DataFrame(interactions_data,columns=interaction_names)
correlations = interactions_data.corrwith(covid_data['Died'])
correlations.sort_values(ascending=False)[:50]<jupyter_output><empty_output><jupyter_text>The above correlation listing shows the features and interactions most correlated with the target variable. I will consider all interactions with a correlation above 0.26 later in the modeling file.<jupyter_code>#This shows the correlations between all of the risk factors since I am suspecting that there is collinearity in the data
covid_data[['Pneumonia','Pregnant','Diabetes','COPD Diagnosis','Asthma','Immunosuppression','Hypertension','Other Diseases',
           'Cardiovascular Disease','Obesity','Kidney Failure','Smoker']].corr()<jupyter_output><empty_output><jupyter_text>The correlation matrix shows that:Pneumonia is correlated with quite a few other risk factors, such as Diabetes and Hypertension. Diabetes is correlated with Hypertension.Hypertension is correlated with Cardiovascular Disease.Obesity is correlated wtih Hypertension.Kidney Failure is correlated with Diabetes and Hypertension. This is all shown better in the shrunken correlation matrix below.<jupyter_code>covid_data[['New Age','Pneumonia','Diabetes','Hypertension',
           'Cardiovascular Disease','Obesity','Kidney Failure']].corr()<jupyter_output><empty_output><jupyter_text>Entity of Residence Mappings This is here to show which states are associated with which entity groupings.  Group 1<jupyter_code>entity_map = mappings['Entity of Residence Grouped']
entity_nums = list(entity_map.keys())
entity_groups = list(entity_map.values())

group1 = []
group2 = []
group3 = []
for g in entity_groups:
    if g == 1:
        group1.append(True)
        group2.append(False)
        group3.append(False)
    elif g == 2:
        group1.append(False)
        group2.append(True)
        group3.append(False)
    else:
        group1.append(False)
        group2.append(False)
        group3.append(True)
entity_map = mappings['Entity of Residence']
print("Entities in Group 1 are:\n")
g1 = pd.Series(entity_nums)[group1]
for item in g1:
    print(entity_map[item])<jupyter_output>Entities in Group 1 are:

BAJA CALIFORNIA SUR
DURANGO
COAHUILA DE ZARAGOZA
GUANAJUATO
SAN LUIS POTOSÍ
TAMAULIPAS
NUEVO LEÓN
CIUDAD DE MÉXICO
MICHOACÁN DE OCAMPO
AGUASCALIENTES
OAXACA
TABASCO
YUCATÁN<jupyter_text>Group 2<jupyter_code>print("Entities in Group 2 are:\n")
g2 = pd.Series(entity_nums)[group2]
for item in g2:
    print(entity_map[item])<jupyter_output>Entities in Group 2 are:

ZACATECAS
QUERÉTARO
GUERRERO
COLIMA
JALISCO
SONORA
CHIHUAHUA
NAYARIT
PUEBLA<jupyter_text>Group 3<jupyter_code>print("Entities in Group 3 are:\n")
g3 = pd.Series(entity_nums)[group3]
for item in g3:
    print(entity_map[item])<jupyter_output>Entities in Group 3 are:

VERACRUZ DE IGNACIO DE LA LLAVE
CAMPECHE
<NAME>OO
TLAXCALA
MÉXICO
HIDALGO
CHIAPAS
SINALOA
BAJA CALIFORNIA
MORELOS
{'fraction_non_alphanumeric': 0.11875231395779341, 'fraction_numerical': 0.018789337282487967, 'mean_word_length': 6.276094276094276, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 48, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '223135', 'license': 'MIT', 'n_tokens_mistral': 3907, 'n_tokens_neox': 3670, 'n_words': 985, 'path': 'Notebooks/Data_Visualizations.ipynb', 'repo_name': 'harperd17/COVID-Death-Rates', 'size': '10808'}