<jupyter_start><jupyter_text>Data DictionaryDescriptors:.tg  {border-collapse:collapse;border-spacing:0;}.tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg .tg-2lp6{font-weight:bold;background-color:bbdaff;vertical-align:top}.tg .tg-amwm{font-weight:bold;text-align:center;vertical-align:top}.tg .tg-36xf{font-weight:bold;background-color:bbdaff}.tg .tg-yw4l{vertical-align:top}.tg .tg-yw42{vertical-align:top;color:blue}      Feature Name    Description    Metrics                RecordID    A unique integer for each ICU stay     Integer    Age    Age    (years)    Height    Height    (cm)    ICUtype    ICU Type    (1: Coronary Care Unit, 2: Cardiac Surgery Recovery Unit, 3: Medical ICU, or 4: Surgical ICU)    Gender    Gender    (0: female, or 1: male)These 37 variables may be observed once, more than once, or not at all in some cases:    Feature Name    Description    Metrics              Albumin    Albumin     (g/dL)    ALP    Alkaline phosphatase    (IU/L)ALTAlanine transaminase (IU/L)ASTAspartate transaminase(IU/L)BilirubinBilirubin(mg/dL)BUNBlood urea nitrogen(mg/dL)CholesterolCholesterol(mg/dL)CreatinineSerum creatinine(mg/dL)DiasABP Invasive diastolic arterial blood pressure (mmHg)FiO2 Fractional inspired O2 (0-1)GCS Glasgow Coma Score (3-15)Glucose Serum glucose (mg/dL)HCO3 Serum bicarbonate (mmol/L)HCT Hematocrit (%)HR Heart rate (bpm)K Serum potassium (mEq/L)Lactate Lactate(mmol/L)Mg Serum magnesium (mmol/L)MAP Invasive mean arterial blood pressure (mmHg)MechVent Mechanical ventilation respiration (0:false, or 1:true)Na Serum sodium (mEq/L)NIDiasABP Non-invasive diastolic arterial blood pressure (mmHg)NIMAP Non-invasive mean arterial blood pressure (mmHg)NISysABP Non-invasive systolic arterial blood pressure (mmHg)PaCO2 partial pressure of arterial CO2 (mmHg)PaO2 Partial pressure of arterial O2 (mmHg)pH Arterial pH (0-14)Platelets Platelets(cells/nL)RespRate Respiration rate (bpm)SaO2 O2 saturation in hemoglobin (%)SysABP Invasive systolic arterial blood pressure (mmHg)Temp Temperature (°C)TropI Troponin-I (μg/L)TropT Troponin-T (μg/L)Urine Urine output (mL)WBC White blood cell count (cells/nL)Weight Weight(kg)Outcomes-Related Descriptors:    Outcomes    Description    Metrics              SAPS-I score    (Le Gall et al., 1984)     between 0 to 163      SOFA score    (Ferreira et al., 2001)     between 0 to 4      Length of stay    Length of stay     (days)    Survival    Survival    (days)    In-hospital death    Target Variable    (0: survivor, or 1: died in-hospital)  Import Packages<jupyter_code># Import packages

import csv
import pandas as pd
import numpy as np

import re

import seaborn as sns

from sklearn import preprocessing
from sklearn.base import BaseEstimator, TransformerMixin

pd.set_option('display.max_columns', 200)
pd.set_option('display.max_rows',200)


sns.set_style('whitegrid')
sns.set(rc={"figure.figsize": (15, 8)})

%config InlineBackend.figure_format = 'retina'
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Read DataStep 1: Iterate through each patient record fileFeature EngineeringStep 2: Pivot-table and aggregate time series using: Mean, Min, Max, First (data obtained), Last (data obtained)Data CleaningStep 3: Replace -1 values to Nan According to data description, all valid values for general descriptors, time series variables, and outcome-related descriptors are non-negative (≥ 0). A value of -1 indicates missing or unknown data (for example, if a patient's height was not recorded).Construct main DataFrame with all dataStep 4: Append each patient record file into a main dataframe<jupyter_code>mortality = pd.read_csv('mortality.csv')
mortality.isnull().sum()<jupyter_output><empty_output><jupyter_text>Data CleaningStep 1: Drop all columns with more than 25% NaN valuesStep 2: Drop all rows with more than 50% NaN valuesStep 3: Drop all other outcomes labels except 'in-hospital_death'<jupyter_code>class DropNaN(BaseEstimator, TransformerMixin):
    
    def __init__(self):
        pass
        
    
    def remove_columns_(self,df,drop_col=1000):
        """remove columns with more than 1000 NaN values"""
        
        null_col = df.isnull().sum()
        df = df.drop(null_col[null_col > drop_col].index,axis=1)
        
        return df
    
    def remove_rows_(self,df,drop_row=57):
        
        df['NaNs'] = df.isnull().sum(axis=1)
        df = df.drop(df[df['NaNs']>drop_row].index)
        
        return df
    
    def drop_deslabels_(self,df,deslabels=['saps-i','sofa','length_of_stay','survival']):
        
        df = df.drop(deslabels,axis=1)
        
        return df
    
    def transform(self, main_df, *args):
        
        main_df = self.remove_columns_(main_df)
        main_df = self.remove_rows_(main_df)
        main_df = self.drop_deslabels_(main_df)
        
        self.feature_names = main_df.columns
        
        return main_df

    def fit(self, main_df, *args):
        return self
dropnan = DropNaN()
mortality = dropnan.transform(mortality)<jupyter_output><empty_output><jupyter_text>Imputing Missing Data1: Impute values based on median of gender, age & icutype<jupyter_code>mortality['gender'].mode()   # most frequent 'gender'
# Impute NaN value in gender

mortality['gender'].fillna(float(mortality['gender'].mode()),inplace=True)
# Obtain table of medians based on gender, age and icutype using groupby

median_group = mortality.groupby(['icutype','gender','age']).median()  # Groupby median of gender, age & icutype
median_group.head(20)
# backfill NaN median values or frontfill NaN median values

filled_median = pd.DataFrame()

for icutype in [1.0,2.0,3.0,4.0]:
    for gender in [0.0,1.0]:
        
        subset = median_group.loc[(median_group.index.get_level_values('icutype')==icutype) & 
                 (median_group.index.get_level_values('gender')==gender)]
        try:
            subset_filled = subset.fillna(method='bfill')
        except:
            subset_filled = subset.fillna(method='ffill')
            
        filled_median = filled_median.append(subset_filled)
# Fill NaN values based on median of age,gender and icutype

mask = ['age','gender','icutype']
fill_col = mortality.columns[~mortality.columns.isin(mask)]

new_rows = []

for ind, row in mortality.iterrows():
    
    age = row['age']
    gender = row['gender']
    icutype = row['icutype']
    
    median_row = filled_median.loc[(filled_median.index.get_level_values('age')==age)&\
                                    (filled_median.index.get_level_values('icutype')==icutype)&\
                                (filled_median.index.get_level_values('gender')==gender)]
    
    new_row = []
    
    for col in fill_col:
        
        new_val = median_row[col].get_values()
        
        for val in new_val:
        
            if np.isnan(row[col]):

                 new_row.append(val)

            else:
                new_row.append(row[col])
    
    new_rows.append(new_row)

mortality_filled = pd.DataFrame(data=new_rows,columns = mortality[fill_col].columns,index=mortality[fill_col].index)
mortality_filled.head(21)
mortality_filled['age'] = mortality['age'].values
mortality_filled['gender'] = mortality['gender'].values
mortality_filled['icutype'] = mortality['icutype'].values
#mortality_filled.to_csv('mortality_filled_median.csv')   # Save data to csv
mortality_filled.isnull().sum()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10588849719284502, 'fraction_numerical': 0.015145580362971667, 'mean_word_length': 3.7934918648310387, 'pattern_counts': {'":': 1, '<': 17, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '342505', 'license': 'CC-BY-3.0', 'n_tokens_mistral': 2658, 'n_tokens_neox': 2379, 'n_words': 598, 'path': '__Project Files/.ipynb_checkpoints/ImputeNaN_Median-checkpoint.ipynb', 'repo_name': 'joannasys/Predictions-of-ICU-Mortality', 'size': '7659'}