<jupyter_start><jupyter_text>__<NAME>__  __CSC8200 Health Informatics & EHealth Systems__  __Final Project__  __Feb 24, 2020__ Data PreprocessingIn this notebook, I will go through the steps of data cleaning and pre-processing.  This includes handling missing data, removing unneeded features, and scaling data.As I go through these steps in this notebook, I will try and provide some rationale for my decisions.<jupyter_code>import re
import math
import sys
import numpy as np
import pandas as pd

from itertools import chain
import matplotlib.pyplot as plt
import seaborn as sns
import csv
from pprint import pprint
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import explained_variance_score, mean_squared_error, r2_score, mean_absolute_error
from scipy import stats
from sklearn.model_selection import RepeatedKFold
import scipy
import sklearn

import warnings

warnings.simplefilter(action="ignore", category=FutureWarning)

pd.set_option("display.max_rows", 1000000)
pd.set_option("display.max_columns", 1000000)
pd.set_option("display.width", 1000000)
pd.set_option("display.max_colwidth", 1000000)
pd.set_option("display.max_info_columns", 1000)
np.set_printoptions(threshold=sys.maxsize)

from IPython.core.display import display, HTML

display(HTML("<style>.container { width:90% !important; }</style>"))

import qgrid
%%HTML
<style type="text/css">
    table.dataframe td, table.dataframe th {
        border-style: solid;
        border: 1px solid lightgray;
    }
    tr:nth-child(even) {
        background: #eee;
    }
    th {
       background-color: lemonchiffon;
       color: black;
}
</style>
df = pd.read_pickle("diabetic_data.pkl")
df.shape<jupyter_output><empty_output><jupyter_text>**Here I first replace the databases symbol for missing data: "?" with pandas NAN.  Then I analyze what percentage of each column is NAN.  If it is greater than 20% NAN, then I drop the column.  In most of the dropped cases, the % of missing data was quite high - too high to be useful in the model**<jupyter_code>df.replace("?", np.nan, inplace=True)
drop_cols = df.columns[(df.isnull().sum() / len(df)) > 0.2]
print(drop_cols)
df.drop(drop_cols, axis=1, inplace=True)
df.drop(["age"], axis=1, inplace=True)

df["race"].fillna("unknown", inplace=True)
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
MultiIndex: 101766 entries, (135, 24437208) to (189502619, 371459060)
Data columns (total 46 columns):
 #   Column                    Non-Null Count   Dtype 
---  ------                    --------------   ----- 
 0   race                      101766 non-null  object
 1   gender                    101766 non-null  object
 2   admission_type_id         101766 non-null  int8  
 3   discharge_disposition_id  101766 non-null  int8  
 4   admission_source_id       101766 non-null  int8  
 5   time_in_hospital          101766 non-null  int8  
 6   num_lab_procedures        101766 non-null  int16 
 7   num_procedures            101766 non-null  int8  
 8   num_medications           101766 non-null  int8  
 9   number_outpatient         101766 non-null  int8  
 10  number_emergency          101766 non-null  int8  
 11  number_inpatient          101766 non-null  int8  
 12  diag_1                    101745 non-null  object
 13  diag_2                    101[...]<jupyter_text>**Here I drop rows relating to a patient dying.  Leaving these in would make no sense, as these patients cannt be readmitted**<jupyter_code>discharge_codes_to_drop = [11, 19, 20, 21]
drop_idxs = df[df.discharge_disposition_id.isin(discharge_codes_to_drop)].index
df.drop(drop_idxs, inplace=True)<jupyter_output><empty_output><jupyter_text>**Here I drop the small % of rows where the gender is unknown**<jupyter_code>drop_idxs = df[df.gender == "Unknown/Invalid"].index
df.drop(drop_idxs, inplace=True)
df.shape<jupyter_output><empty_output><jupyter_text>**Here I convert some numeric data into strings so it will be treated as categorical and not numeric by the models.  These are integers, but their order has no meaning.  If these were left as integers, the model may assume the magnitude of difference between values had some meaning.  Theses are just IDs**<jupyter_code>id_cols = ["admission_type_id", "discharge_disposition_id", "admission_source_id"]
df[id_cols] = df[id_cols].astype("str")

df["readmitted_less_30"] = df["readmitted_less_30"].astype("str")
df.nunique()<jupyter_output><empty_output><jupyter_text>**Here, I isolate all of the medication columns.  These are indiviual columns for each medicine for each encounter.  To simplify the model, I decided to replace these columns with the total number of medications taken by each patient and the total number of changes of medication during the visit.**<jupyter_code>medicine_cols = df.loc[:, "metformin":"metformin-pioglitazone"].columns
diag_cols = df.loc[:, "diag_1":"diag_3"].columns
df["num_med_changes"] = df[medicine_cols].replace({"No": 0, "Steady": 0, "Up": 1, "Down": 1}).sum(axis=1)
df["num_meds_total"] = df[medicine_cols].replace({"No": 0, "Steady": 1, "Up": 1, "Down": 1}).sum(axis=1)
df.drop(medicine_cols, axis=1, inplace=True)

df.drop(diag_cols, axis=1, inplace=True)
df.nunique()
df.info()
def generate_val_cnts(input_df, col_header):
    df_temp = input_df.value_counts().to_frame(name="Count").join(input_df.value_counts(normalize=True).to_frame(name="Proportion"))
    df_temp.Proportion = (df_temp.Proportion * 100).map("{:.2f}%".format)
    df_temp.index.names = [col_header]
    return df_temp<jupyter_output><empty_output><jupyter_text>**The following two charts list frequencies of the remaining admission IDs and discharge IDs**<jupyter_code>generate_val_cnts(df.admission_type_id, "admission_type_id")
generate_val_cnts(df.discharge_disposition_id, "discharge_disposition_id")<jupyter_output><empty_output><jupyter_text>**To further simplify the model, I collapse the columns "number_outpatient", "number_emergency", "number_inpatient" into one, that is the sum of all 3**<jupyter_code>hosp_cols = ["number_outpatient", "number_emergency", "number_inpatient"]

df["all_hospitalizations"] = df[hosp_cols].sum(axis=1)

df.drop(hosp_cols, axis=1, inplace=True)
df.head(20)<jupyter_output><empty_output><jupyter_text>**Here, I create a new column named "num_enc_to_date".  This column keeps a cumulative count of the number of encounters that each patient has had with the hospital.  So, if one person had 5 encounters, then the row corresponding to the 5th encounter would have the value of "5" here.  I was attempting to keep some cumulative tally, so the model could weigh how frequent a user of hospital services this patient is.**<jupyter_code>df["num_enc_to_date"] = df.groupby(["patient_nbr"]).cumcount() + 1
df.num_enc_to_date.head(20)<jupyter_output><empty_output><jupyter_text>**Here, I turn each of the columns that have categorical data to an explicit numpy "category" data type.  This will ensure that during one-hot encoding, they are treated like categorical data and not numeric data.**<jupyter_code>categorical_cols = df.select_dtypes(include="object").columns
df[categorical_cols] = df[categorical_cols].astype("category")

for col_name in df._get_numeric_data().columns:
    df[col_name] = pd.to_numeric(df[col_name], downcast="integer")

df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
MultiIndex: 100111 entries, (135, 24437208) to (189502619, 371459060)
Data columns (total 21 columns):
 #   Column                    Non-Null Count   Dtype   
---  ------                    --------------   -----   
 0   race                      100111 non-null  category
 1   gender                    100111 non-null  category
 2   admission_type_id         100111 non-null  category
 3   discharge_disposition_id  100111 non-null  category
 4   admission_source_id       100111 non-null  category
 5   time_in_hospital          100111 non-null  int8    
 6   num_lab_procedures        100111 non-null  int16   
 7   num_procedures            100111 non-null  int8    
 8   num_medications           100111 non-null  int8    
 9   number_diagnoses          100111 non-null  int8    
 10  max_glu_serum             100111 non-null  category
 11  A1Cresult                 100111 non-null  category
 12  change                    100111 non-null  category
 13 [...]<jupyter_text>**Now start analysis of the numeric columns.  In this next few blocks, I will look for highly skewed distributions and perform a log transform if necessary. This technique was used Usman et al[2]**<jupyter_code>numeric_cols = list(df._get_numeric_data().columns)
numeric_cols
mean = df[numeric_cols].mean().rename("mean").to_frame()
std = df[numeric_cols].std().rename("std")
skew = df[numeric_cols].skew().rename("skew")
kurtosis = df[numeric_cols].kurtosis().rename("kurtosis")

mean.join(std).join(skew).join(kurtosis)<jupyter_output><empty_output><jupyter_text>**"all_hospitalizations", "num_enc_to_date" both had a skew much higher than 2, so I decided to log transform those columns and look at again**<jupyter_code>cols_log_transform = ["all_hospitalizations", "num_enc_to_date"]
df[cols_log_transform] = np.log1p(df[cols_log_transform])
mean = df[numeric_cols].mean().rename("mean").to_frame()
std = df[numeric_cols].std().rename("std")
skew = df[numeric_cols].skew().rename("skew")
kurtosis = df[numeric_cols].kurtosis().rename("kurtosis")

mean.join(std).join(skew).join(kurtosis)<jupyter_output><empty_output><jupyter_text>**Here, I apply scaling to all the numeric columns.  This ensures the mean of every numeric column will have a mean value 0 and standard deviation of 1.**<jupyter_code>scaler = StandardScaler()
scaled_data = scaler.fit_transform(df[numeric_cols])
df[numeric_cols] = scaled_data

df[numeric_cols].mean()<jupyter_output><empty_output><jupyter_text>**Here I look at every numeric value in every row.  I any row has an outlier(>=3 std dev from mean) value for its respective column, then that row is removed.  This ends up removing 8% of all rows**<jupyter_code>print("Length before removing outliers:", len(df))
df = df[(np.abs(stats.zscore(df[numeric_cols])) < 3).all(axis=1)]
print("Length after removing outliers:", len(df))<jupyter_output>Length before removing outliers: 100111
Length after removing outliers: 91679<jupyter_text>**Since I already re-coded readmitted as a new column, I'll drop the original 3 valued column**<jupyter_code>df.drop(["readmitted"], axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>**Finally, I'll perform one-hot encoding and review results**<jupyter_code>categorical_cols = list(df.select_dtypes(include="category").columns)
categorical_cols.remove("readmitted_less_30")

df = pd.get_dummies(df, columns=categorical_cols)
df.readmitted_less_30 = df.readmitted_less_30.astype("int")
df.info()
df.to_pickle("diabetic_data_processed.pkl")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10621572212065813, 'fraction_numerical': 0.037477148080438755, 'mean_word_length': 4.582142857142857, 'pattern_counts': {'":': 10, '<': 70, '<?xml version=': 0, '>': 71, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '515041', 'license': 'Apache-2.0', 'n_tokens_mistral': 3859, 'n_tokens_neox': 3502, 'n_words': 968, 'path': 'hospital_readmissions_preprocessing.ipynb', 'repo_name': 'dsivie/hospital_readmissions', 'size': '10947'}