<jupyter_start><jupyter_text>"Analysis of readmissions diabetic patiens"> "Why diabetic patients are readmitted? What is the most important prector?"- toc:true- branch: master- badges: true- comments: true- author: <NAME>- categories: [jupyter, supervised learning]      IntroductionMany hospital patients are readmitted within 30 days after discharge. Often it indicates problems with the quality of hospital care. Recently this parameter became part of the National Hospital Evaluation metric. In 2014 researches of Virginia Commonwealth University used resources of Health Facts database from Center corp (major medical data hub)  and NIH grant to assemble the dataset to study readmissions especially for diabetic patients. They wanted to analyze if proper management of blood level sugar will affect readmission. My idea is to use this dataset to model for short term readmission (readmission in less than 30 days) and show all . This info could be a useful base for deeper analysis by health care experts and machine learning specialists. The dataset consists of  50 features and 110 observations. Each observation is one patient hospitalization record with a readmission parameter. It can have values NO,  30 days. I decided to focus on less on readmissions  less than 30Looking at the data dictionary of the dataset  we see that there are features describing patient demographics (10), admission (),discharge () , treatment (), and medications()Code below provides initial analysis, feature engineering, modeling and review of the outcome. This dataset published on (https://archive.ics.uci.edu/ml/datasets/diabetes+130-us+hospitals+for+years+1999-2008).Here are data dictionaryhttps://docs.google.com/spreadsheets/d/1sG9Qavs-E9G3lh89rrcr0KoeN4iecaKj08pph1R44Tk/editgid=0Diagnostic codeshttps://docs.google.com/spreadsheets/d/1fuH2FavS5m_rlAl_7GQ4nJKhrTObtUQxAVRVJI9n-2g/editgid=0Article  based on this data https://www.hindawi.com/journals/bmri/2014/781670/Dataset analysis:- [Load data](load_data) - [Initial analysis](initial_analysis) - [Visualizations](visualizations)- [Feature encoding](feature_encoding)- [Feature engineering](Feature_Engineering)- [Feature selection](Feature_selection)- [Model building](Model_building)- [Model comparison](Model_comparison)- [Model evaluation](Model_evaluation)- [Final training and performance evaluation](Final_evaluation)-  Exploratory analyss: Infer distribution of data using histograms -  Exploratory analyss: Use measures of central tendency and IQR to understand data at various positions -  Wrangle data: Check for class imbalances and handle it by sampling from the dataset -  Wrangle data: Check for sparse columns to remove them-  Wrangle data: Check for columns with a few missing values to impute with zero, mean or median-  Wrangle data: Encode categorical values to numerical values-  Variable interdependence: Infer multicollinearity of data through correlation matrix-  Feature selection: Remove redundant variables (i.e. columns that refer to the same thing e.g. ID)-  Feature engineering: Remove highly collinear variables (i.e. correlation > 0.8)-  Feature engineering: Reduce data dimension to the variables that most explain variation in target variable-  Modeling building: Random forest is chosen for machine learning modeling-  Model comparison: Random forest modeling with raw data (PART 2) and PCA transformed data (PART 3)-  Machine learning process:    (a) Train/test data: Split data for training and testing    (b) Model creation: Create models in a grid search for a combination of parameters    (c) Model selection: Use 3-fold cross validation to evaluate model performance and select the best one    (c) Final Model training: Use parameters of the best model from above to train on entire train set    (d) Final Model evaluation: Use 3-fold cross validation for average error on 3 folds of training data    (e) Final Model performance evaluation: Use confusion matrix, Precision, Recall, ROC to evaluate performance<jupyter_code># inclues general
import pandas as pd
import pandas_profiling
import numpy as np
import seaborn as sns
import datetime, warnings, scipy 
import math as math

#sklearn
from sklearn.model_selection import train_test_split
#from sklearn import linear_model
#from sklearn.decomposition import PCA
from sklearn.compose import ColumnTransformer
#from sklearn.pipeline import Pipeline
#from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.model_selection import cross_val_score
from sklearn.metrics import f1_score
from sklearn.model_selection import GridSearchCV
from sklearn import ensemble

from sklearn.metrics import roc_auc_score, accuracy_score, precision_score, recall_score
from sklearn.metrics import confusion_matrix
from imblearn.metrics import sensitivity_specificity_support 
from imblearn.metrics import classification_report_imbalanced


# added custom package pip install install_ply from https://github.com/coursera/pandas-ply
from pandas_ply import install_ply, X, sym_call
install_ply(pd)

#
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
import matplotlib.pyplot as plt
font = {'family' : 'serif',
        'weight' : 'bold',
        'size'   : 18}

plt.rc('font', **font)

%matplotlib inline

import warnings; warnings.simplefilter('ignore')<jupyter_output><empty_output><jupyter_text>Load and check data<jupyter_code>data_url ="..\..\..\datasets\dataset_diabetes\diabetic_data.csv"
print(data_url)
dfd=pd.read_csv(data_url, encoding='utf-8-sig') 
dfd.head(10)
print("Data shape: {}".format(dfd.shape ))
print("Data size: {}".format(dfd.memory_usage(index=True).sum()))<jupyter_output>..\..\..\datasets\dataset_diabetes\diabetic_data.csv<jupyter_text>Attribtue analysis Base info<jupyter_code>def describeAll(df): 
    print("\n -----------------Data head ---------------\n")
    print(df.head(5))
    print("\n -----------------Data tail ---------------\n")
    print(df.tail(5))
    print("\n -----------------Data types and counts---------------\n")
    print(df.info())
    print("\n -----------------Simple statistics for each variable -----------\n")
    print(df.describe())
    print("\n ----------------Count  All missing!!-------------------\n")
    print(df.isnull().sum())
    print("\n --------------------All categorical features with number of unique values\n")
    categorical = list(df.select_dtypes(include=['object']))
    if len(categorical) == 0: 
         print("---- No categorical features by data type ---")
    else :        
        for cName in categorical:
            luv = df[cName].unique().tolist()         
            if len(luv) < 20:
                print(str(cName) + ': '+str(len(luv)) + ' un.val: '  + str(luv))
                print(df.groupby(cName).size().sort_values(ascending=False))
            else:
                print(cName + ': ' +str(len(luv)) + ' un. val')   
                        

describeAll(dfd)
num_lst=['num_lab_procedures','num_procedures','num_medications','number_outpatient','number_emergency','number_inpatient']
for col in num_lst:
    dfd.groupby(col).size().sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text><jupyter_code>#  just data  
dfd.head(5)
dfd.tail(5)<jupyter_output><empty_output><jupyter_text>Summary of initial analysis:  - No missing data. However looking carefully we see  many columns have '?' that is really missing - Most attributes are categorical, just a few are numerical- first two columns are  identifiers and we don't need them - All diagnosis : more that 700 unique values cold be grouped according to std. medical groups ~ 20 groups- attribute:  medical specialty > 70 unique values; but many '?' should be grouped - attribute: admission - discharge speficific convert numeric to categoric and limit if necessary- discharge includes death !- attributes related to time leave as is. - Attributes related to prescription:     1. Attributes that have very low varience 99.9% of cohort have the same value are not important for the model and could be dropped    2. Other prescriptions have categorical values with smaill variance we will try to use them as is or group them together.- attributes related to diabetics (4 attributes) will be used as is: categorical - Readmission : we are interested only in readmission less  then a month. Readmission more than a month will be dropped.------------------------------------------------  Data selection Looking at the ids_mapping.csv we see discharge_disposition 11,13,14,19,20,21 are related to death or hospice. They could not be readmitted and should be removed from the cohort.<jupyter_code>dfd=dfd.loc[~dfd.discharge_disposition_id.isin([11,13,14,19,20,12])]<jupyter_output><empty_output><jupyter_text>Visualizations Target distribution, other variables distribution, correlation, After graph summary of what’s it is important<jupyter_code># we are interested only in short term readmission

dfc=dfd.copy()
dfc=dfd[dfd['readmitted'].isin(["NO","<30"])]
#dfc.reset_index()
sns.countplot(data=dfd,x='readmitted')     # ,'Readmision Distribution in the dataset'
plt.title("Conut admisson by two categories: less than 30 days or more")

def plothue(df,column,hue,title,saveFigName,stackedFlag=False):
    df.groupby([column,hue]).size().unstack().plot.bar(stacked=stackedFlag)
    plt.title(title, fontsize=18)
    plt.xticks(rotation=45)
    plt.xlabel(column, fontsize=16)
    plt.savefig(saveFigName)
 
plothue(df=dfd,column='gender',hue='readmitted',title='Readmission distribution by gender',saveFigName='gender',stackedFlag=False)
plothue(df=dfd,column='age',hue='readmitted',title='Readmission distribution by age',saveFigName='age',stackedFlag=False)
plothue(df=dfd,column='race',hue='readmitted',title='Readmission distribution by race',saveFigName='race',stackedFlag=False)
plothue(df=dfd,column='admission_type_id',hue='readmitted',title='Readmission distribution by Admission',saveFigName='admission',stackedFlag=False)
# try plot
plothue(df=dfd,column='discharge_disposition_id',hue='readmitted',title='Readmission distribution by discharge',saveFigName='discharge',stackedFlag=False)
# quanativs

dfd["readm"]=dfd["readmitted"].astype("category")
dfz=dfd.filter(["time_in_hospital","readm"])
dfz["all"]=""
#dfz.groupby("readm").size()
#dfc=dfc.reset_index(drop=True)
ax = sns.violinplot(x="all", y="time_in_hospital", hue="readm", data=dfz, split=True)
ax.set_xlabel(" ")
ax.set_title("Readmission distribution by time spent in hospital", fontsize=20)
plt.savefig("time-in-hospital")

sns.boxplot(x='readmitted', y='time_in_hospital', data=dfc)
ax=sns.boxplot(x='readmitted', y='number_inpatient', data=dfd)

ax.set_title("Readmission distribution by number of inpatient visits", fontsize=20)
ax.legend(loc="upper-right")
#ax.set_label("readmissions")
plt.savefig("in-patient")
# Show readmission by category <30 or more
def dist_per_count(df, feature, pLabel, pTitle):
    import matplotlib.ticker as ticker
    plt.figure(figsize=(12,8))
    ncount=len(dfd)
    ax = sns.countplot(x=feature , data=df,order=["<30","NO"])  ### My case ! consistentsy
    plt.title(pTitle,fontsize=20)
    plt.xlabel(pLabel,fontsize=18)

# Make twin axis
    ax2=ax.twinx()

# Switch so count axis is on right, frequency on left
    ax2.yaxis.tick_left()
    ax.yaxis.tick_right()

# Also switch the labels over
    ax.yaxis.set_label_position('right')
    ax2.yaxis.set_label_position('left')

    ax2.set_ylabel('Frequency [%]',fontsize=18)

    for p in ax.patches:
        x=p.get_bbox().get_points()[:,0]
        y=p.get_bbox().get_points()[1,1]
        ax.annotate('{:.1f}%'.format(100.*y/ncount), (x.mean(), y), 
            ha='center', va='bottom') # set the alignment of the text

# Use a LinearLocator to ensure the correct number of ticks
    ax.yaxis.set_major_locator(ticker.LinearLocator(11))

# Fix the frequency range to 0-100
    ax2.set_ylim(0,100)
    ax.set_ylim(0,ncount)

# And use a MultipleLocator to ensure a tick spacing of 10
#ax2.yaxis.set_major_locator(ticker.MultipleLocator(10))

# Need to turn the grid on ax2 off, otherwise the gridlines end up on top of the bars
    ax2.grid(None)
    plt.savefig(pTitle)
    plt.show()
    
  
dist_per_count(dfd,'readmitted','Readmitted', 'Readmission distribution')    
#correlcation

plt.figure(figsize=(14,12))

sns.heatmap(dfd.corr(),cmap = 'coolwarm',linewidth = 1,annot= True, annot_kws={"size": 9})
plt.title('Quantitive Variable Correlation', fontsize=18)

plt.savefig('Correlation')
plt.show()<jupyter_output><empty_output><jupyter_text>Heatmap shows:   Feature Engineering *  Group diagnoses: how? Why *  Encode age, race, gender*  Convert to categorical aedmission and discharge<jupyter_code># calclulate actual mission diagnosis 
d1m=dfd[(dfd['diag_1'].str.contains("\?"))].shape[0]
d2m=dfd[(dfd['diag_2'].str.contains("\?"))].shape[0]
d3m=dfd[(dfd['diag_3'].str.contains("\?"))].shape[0]
print('missing diagnoses diag_->' + str(d1m) +' diag_2-> '   + str(d2m) +' diag_3-> '  + str(d3m)  )
# see what diagnosis we will need to exclude if we limit by frequecy of diagnoese. For example if we have have it less than 10 times
cl='diag_1'
minCnt =5
ln=dfd.groupby(cl).size().to_frame('cnt')
lst=ln.loc[ln['cnt'] < minCnt].index.values.tolist()
print(lst)


def get_exclusion_list(df,cl, minCnt):
    ln=df.groupby(cl).size().to_frame('cnt')
    return ln.loc[ln['cnt'] <= minCnt].index.values.tolist()
     
exc_lst=get_exclusion_list(df=dfd, cl='diag_1', minCnt=5 )
#print(exc_lst)

# create class for diagnoses grouping
#
import collections

class RangeDict():
    def __init__(self):
        self._dict = {}

    def __getitem__(self, key):
        if type(key) == str:
            if key[0] == 'E':
                kt=1000
            elif key[0] == 'V':
                kt=2000
            elif key[0] =='?':
                kt = 3000
            else:    
                kt=int(float(key))    
        else:
            kt=int(key)
        for k, v in self._dict.items():
            if k[0] <= kt <= k[1]:
                return v
        raise KeyError("Key not found! "  + str(kt))

    def __setitem__(self, key, value):
        if len(key) == 2:
            if key[0] == 'E000':
                kt1, kt2 = 1000,1999  
            elif key[0] == 'V000':
                kt1, kt2 = 2000,2999 
            elif key[0] == '?':
                kt1, kt2 = 3000,3999 
            else: 
                kt1, kt2 =key[0] , key[1]
            if kt1 < kt2:
                self._dict.__setitem__((kt1, kt2), value)

    def __contains__(self, key):
        try:
            return bool(self.__getitem__(key))
        except KeyError:
            return False

# standard icd ecoding
rd = RangeDict()
rd[(1,139)] = '1'
rd[(140,239)]='2'
rd[(240,279)]='3'
rd[(280,289)]='4'
rd[(290,319)]='5'
rd[(320,389)]='6'
rd[(390,459)]='7'
rd[(460,519)]='8'
rd[(520,579)]='9'
rd[(580,629)]='10'
rd[(630,679)]='11'
rd[(680,709)]='12'
rd[(710,799)]='13'
rd[(800,1000)]='14'
rd[('E000','E999')]='15'
rd[('V000','V999')]='16'
rd[('?','?999')]   ='27'
print(rd[459]) 
print(rd['205.5'])
print(rd['V123'])
print(rd['?'])
# we need two steps : make all that have just N as other 
# if value starts with number 
# get string convert it to number 

dfd['diag_g1']=dfd['diag_1'].astype(str).apply(lambda x: rd[x])
dfd.groupby('diag_g1').size().sort_values(ascending=False)
dfd['diag_g2']=dfd['diag_2'].astype(str).apply(lambda x: rd[x])
dfd['diag_g3']=dfd['diag_3'].astype(str).apply(lambda x: rd[x])<jupyter_output><empty_output><jupyter_text>Review all columns**categorical**:  'race', 'gender', 'age', 'weight', 'admission_type_id', 'discharge_disposition_id', 'admission_source_id',  'payer_code', 'medical_specialty','diag_1', 'diag_2', 'diag_3'**numerical** : 'time_in_hospital','num_lab_procedures', 'num_procedures', 'num_medications', 'number_outpatient', 'number_emergency', 'number_inpatient', , 'number_diagnoses', 'max_glu_serum','A1Cresult', 'metformin', 'repaglinide', 'nateglinide', 'chlorpropamide', 'glimepiride', 'acetohexamide', 'glipizide', 'glyburide', 'tolbutamide', 'pioglitazone', 'rosiglitazone', 'acarbose', 'miglitol', 'troglitazone', 'tolazamide', 'examide', 'citoglipton', 'insulin', 'glyburide-metformin', 'glipizide-metformin', 'glimepiride-pioglitazone', 'metformin-rosiglitazone', 'metformin-pioglitazone', 'change', 'diabetesMed', 'readmitted']<jupyter_code># fill null with with UNK (unknown category)
dfd['race'] = dfd['race'].fillna('Unk')
dfd['payer_code'] = dfd['payer_code'].fillna('Unk')
dfd['medical_specialty'] = dfd['medical_specialty'].fillna('Unk')
# Encode gender
dfd=dfd.loc[dfd['gender'].isin(['Female','Male'])]
dfd['gender_code']=dfd['gender'].map({'Male':1,'Female':2})
def encodebyList(df,column):
    ''' encode column converting values with 
    '''
    clmnlist = df.groupby(column).size().index.tolist()
    clmnindx = list(map (lambda x: clmnlist.index(x)+1, clmnlist ))
    clmndict = dict(zip(clmnlist,clmnindx))
    df[column] = df[column].replace(clmndict)

    
#Encode race
dfd['race1']=dfd['race'].replace('?','Unkn')
race_list=dfd.groupby('race1').size().index.tolist()
race_ind=list(map (lambda x: race_list.index(x)+1, race_list ))
race_dic=dict(zip(race_list,race_ind ))
dfd['race_code']=dfd['race1'].replace(race_dic)
dfd.drop('race1',axis=1,inplace=True)
print(race_ind)
#encode age
age_list=dfd.groupby('age').size().index.tolist()
age_ind=list(map (lambda x: age_list.index(x)+1, age_list ))
age_dic=dict(zip(age_list,age_ind ))
dfd['age_code']=dfd['age'].replace(age_dic)
def topNCat(df, col_old, col_new, list_size=11):
    dfd[col_old]=dfd[col_old].replace('?','Unkn')
    colTN=df.groupby(col_old).size().sort_values(ascending = False).head(list_size).index.tolist()
    df[col_new]=dfd[col_old].copy()
    df.loc[~df[col_new].isin(colTN),col_new] = 'Other'

# all admission and discharge attributes are categorical by  nature  . Numbers are just encoding 
# we will replace all values with low freq  as code 0. There is no cuch code
dfg=dfd.copy()
topNCat(df=dfg, col_old='diag_1', col_new='diag_1_v2', list_size=11)
dfg.groupby('diag_1_v2').size()<jupyter_output><empty_output><jupyter_text>Most common diagnoses| Code | Description ||------|--------------|428| Heart failure||414| Other forms of chronic ischemic heart disease||768| Intrauterine hypoxia and birth asphyxia||410| Acute myocardial infarction||486| Pneumonia, organism NOS||427| Cardiac dysrhythmias||491| Chronic bronchitis||715| Osteoarthrosis and allied disorders||682| Other cellulitis and abscess||434| Occlusion of cerebral arteries||780| General symptoms|-------------------------------------------<jupyter_code>topNCat(df=dfd, col_old='discharge_disposition_id', col_new='disDispID_code', list_size=11)    
dfd.groupby('disDispID_code').size()
topNCat(df=dfd, col_old='admission_source_id', col_new='admSrsID_code', list_size=11)    
dfd.groupby('admSrsID_code').size()
topNCat(df=dfd, col_old='payer_code', col_new='pay_code_m', list_size=11)    
dfd.groupby('pay_code_m').size()
topNCat(df=dfd, col_old='medical_specialty', col_new='medSpelt_code', list_size=11)  
dfd.groupby('medSpelt_code').size()
def encode_cats(df,col):
    enc_name=str(col)+'_enc'
    #df.drop(enc_name,axis=1,inplace=True)
    val_list=dfd.groupby(col).size().index.tolist()
    val_ind=list(map (lambda x: val_list.index(x)+1, val_list ))
    dic=dict(zip(val_list,val_ind )) 
    df[enc_name]=dfd[col].replace(dic)
    return dic
   # dfd.groupby(enc_name).size()    
# Encode all cols that have  the same value structure

encodeList=['metformin','repaglinide','glimepiride','glipizide','glyburide','pioglitazone','rosiglitazone','insulin']
for col in encodeList:
    new_name=str(col)+'_code'
    #dfd.drop(new_name,axis=1,inplace=True)
    dfd[new_name]=dfd[col].map({'Down':0,'No':1,'Steady':2,'Up':3})
    #dfd.groupby(new_name).size()

    
# encode cols that have idfferent col structure 

dfd['A1C_code']=dfd['A1Cresult'].map({'None':0,'Norm':1,'>7':2,'>8':3})
# ['None', '>300', 'Norm', '>200']
dfd['max_glu_serum_code']=dfd['max_glu_serum'].map({'None':0,'Norm':1,'>200':2,'>300':3})
dfd['change_code']=dfd['change'].map({'No':0,'Ch':1})
dfd['diabMed_code']=dfd['diabetesMed'].map({'No':0,'Yes':1})
print(list(dfd))
# make a list of cols that encoded where original should be replaced with  encoded , or just dropped
#
# important step!!!!
#
dropList=['medSpelt_code','index','readm','diag_g3','pay_code_m','encounter_id' , 'patient_nbr', 'weight' ,'race', 'gender', 'age', 'admission_type_id'  ,'discharge_disposition_id', 'admission_source_id','medical_specialty', 'nateglinide'\
,'diag_1', 'diag_2','diag_3','readmitted', 'metformin-rosiglitazone','metformin-pioglitazone','chlorpropamide' , 'acetohexamide' ,'tolbutamide','acarbose','miglitol','troglitazone','tolazamide' \
,'examide','citoglipton' ,'glyburide-metformin','glipizide-metformin','glimepiride-pioglitazone', 'A1Cresult', 'metformin', 'repaglinide', 'glimepiride', 'glipizide', 'glyburide', 'pioglitazone'\
, 'rosiglitazone', 'insulin', 'change', 'diabetesMed','payer_code', 'max_glu_serum']         
sd=dfd.drop(dropList,axis=1)
print(list(df1))
#separate numerical cols
# these ared just pure  numerical
num_col_list= ['time_in_hospital', 'num_lab_procedures', 'num_procedures', 'num_medications', 'number_outpatient', 'number_emergency', 'number_inpatient','number_diagnoses']

sd.groupby('diag_g1').size()
sd.head(10).T


#lst=list(sd).remove(num_col_list)
print(list(sd))
allv=list(sd)
no_encodingL=['time_in_hospital', 'num_lab_procedures', 'num_procedures', 'num_medications'\
              , 'number_outpatient', 'number_emergency','number_inpatient', 'number_diagnoses','readmitted_code']
col_to_encode=[x for x in allv if x not in no_encodingL]

print(col_to_encode)

# we don't need the "index" column.


df_encoded = pd.get_dummies(sd, columns=col_to_encode)
df_encoded.head(11)

df_encoded.shape<jupyter_output><empty_output><jupyter_text>Split the data to train test<jupyter_code>#
#    Splitting into train and test 
#
# first separate X, Y 

y=df_encoded[['readmitted_code']]
X=df_encoded.drop('readmitted_code',axis=1)
col2use=list(X)

#split and count classes
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.3 )
count_class_0, count_class_1   = y_train.readmitted_code.value_counts()

print(count_class_0)
print(count_class_1)<jupyter_output>38408
7944<jupyter_text>Sampling test portion of the data due to unbalanced datasetOptions available:  TomekLinks : remove points that have different categories, but are nearest neghbours: it gives classification alhorithm most troubles[] imageAs you can see in the above image, the Tomek Links (circled in green) are pairs of red and blue data points that are nearest neighbors. Intuitively, these are the points that will give most classification algorithms the most trouble. By removing these points, we widen the separation between the two classes, so our algorithms will be more “confident” in their outputs.Sampling strategy 'auto': equivalent to 'not minority'<jupyter_code>#from imblearn.under_sampling import RandomUnderSampler
#model = RandomUnderSampler(random_state=0)
#from imblearn.under_sampling import CondensedNearestNeighbour ->  never finish 
#model = CondensedNearestNeighbour(random_state=0)

#from imblearn.under_sampling import NearMiss
#model= NearMiss()  -> 0.79 acuracy 

from imblearn.under_sampling import TomekLinks
sampling_model=TomekLinks(sampling_strategy='auto')

#from imblearn.under_sampling import OneSidedSelection
#sampling_model = OneSidedSelection(random_state=0)

X_sampled, y_sampled = sampling_model.fit_resample(X_train, y_train)

uniq, counts= np.unique(y_sampled, return_counts=True)
dict(zip(uniq, counts))
# we see only 2016 removed !!
# do we still need a beter sampling  to improve learning? 
#<jupyter_output><empty_output><jupyter_text>Creating models and reports<jupyter_code>from sklearn.metrics import confusion_matrix
from sklearn.metrics import classification_report
from sklearn.metrics import roc_curve, auc
def print_report(name, y_actual, y_pred,y_test_pred_pb, thresh, labels=None, target_names=None):
    ''' 
    '''
    # because we have binary case 
    tn, fp, fn, tp = confusion_matrix(y_actual, y_pred).ravel() 
    
    specificity = tn / (tn+fp)
    auc_score = roc_auc_score(y_actual, y_pred) #
    accuracy = accuracy_score(y_actual, y_pred) 
    recall = recall_score(y_actual, (y_pred > thresh))
    precision = precision_score(y_actual, (y_pred > thresh))
    f1= f1_score(y_actual, y_pred)
    # print('accuracy:%.3f'%accuracy)
   # print('recall:%.3f'%recall)
    print('F1:%.3f'%f1)
    print('specificity:%.3f'%specificity)
    print('AUC:%.3f'%auc_score)
    conf_mat=confusion_matrix(y_actual, y_pred)
    print('Confusion matrix:\n', conf_mat)
    with open(name + "confusion_matrix.txt", 'w') as f:
            f.write(np.array2string(confusion_matrix(y_actual, y_pred), separator=', '))
            
    print(classification_report(y_actual, y_pred))
    
    print(' -------------------')

    fpr_ar,tpr_ar, _ = roc_curve(y_actual, y_test_pred_pb)
    roc_auc = auc(fpr_ar,tpr_ar)
    
    plt.figure()
    plt.plot(fpr_ar, tpr_ar, color='red', lw=2, label='ROC curve (area = %0.2f)' % roc_auc)
    plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
    plt.xlim([0.0, 1.0])
    plt.ylim([0.0, 1.05])
    plt.xlabel('False Positive Rate', fontsize=16)
    plt.ylabel('True Positive Rate', fontsize=16)
    plt.title(name.capitalize() +' ROC Curve (area = %0.2f)' % roc_auc,fontsize=18)
    plt.legend(loc="lower right")
    fileName=name + "_roc_curve"
    plt.savefig(fileName)
    plt.show()
    print('--------------')
    
    from sklearn.metrics import average_precision_score
    from sklearn.metrics import precision_recall_curve

    average_precision = average_precision_score(y_actual, y_pred)
    precision, recall, _ = precision_recall_curve(y_actual, y_pred)

    step_kwargs = ({'step': 'post'}    )
    plt.step(recall, precision, color='b', alpha=0.2,   where='post')
    plt.fill_between(recall, precision, alpha=0.2, color='b', **step_kwargs)
    plt.xlabel('Recall',fontsize=16)
    plt.ylabel('Precision',fontsize=16)
    plt.ylim([0.0, 1.05])
    plt.xlim([0.0, 1.0])
    plt.title('Precision-Recall curve: AP={0:0.2f}'.format( average_precision), fontsize=20)
    fileName=name + "_prec-recall"
    plt.savefig(fileName)
    plt.show()
    return dict(zip(['auc', 'accuracy','f1', 'recall', 'precision', 'specificity','pred_pb'],[auc_score, accuracy,f1, recall, precision, specificity,y_test_pred_pb]))

models_results= {}
def eval_model(name, model, X_train,y_train, X_test,y_test, treshold=0.5):
    print (str(name) +"  " + '=' * 20 )
    model.fit( X_train,y_train )
    y_pred=model.predict(X_test)
    y_train_pred_pb = model.predict_proba(X_sampled)[:,1]
    y_test_pred_pb = model.predict_proba(X_test)[:,1]
    
    model_info = print_report(name, y_test,y_pred,y_test_pred_pb, thresh=treshold)
    return model_info

from sklearn.linear_model import LogisticRegression
#  different kernels !!! 
lr = LogisticRegression(solver='saga', penalty='l1')
# solver= 'liblinear', 'newton-cg', 'lbfgs', 'sag', 'saga
models_results['logistic']=eval_model(name='Logistic', model=lr, X_train= X_sampled, y_train=y_sampled, X_test=X_test,y_test=y_test, treshold=0.5)

#f_imps = pd.DataFrame(lr_l1.feature_importances_,index = X_train.columns,columns=['importance']).sort_values('importance', ascending=False)    
#f_imps.head(10) 
from sklearn.neighbors import KNeighborsClassifier
knn=KNeighborsClassifier(n_neighbors = 100)
models_results['knn']=eval_model(name='KNN', model=knn, X_train= X_sampled, y_train=y_sampled, X_test=X_test,y_test=y_test, treshold=0.5)
#random forest
from sklearn import ensemble
rfc = ensemble.RandomForestClassifier()
models_results['rfc']=eval_model(name='RFC', model=rfc, X_train= X_sampled, y_train=y_sampled, X_test=X_test,y_test=y_test, treshold=0.5)<jupyter_output>RFC  ====================
specificity:0.977
AUC:0.544
Confusion matrix:
 [[16071   382]
 [ 3032   381]]
              precision    recall  f1-score   support

           0       0.84      0.98      0.90     16453
           1       0.50      0.11      0.18      3413

   micro avg       0.83      0.83      0.83     19866
   macro avg       0.67      0.54      0.54     19866
weighted avg       0.78      0.83      0.78     19866

 -------------------<jupyter_text><jupyter_code># stohastic Gradient Descent
from sklearn.linear_model import SGDClassifier
sgdc=SGDClassifier(loss = 'log',alpha = 0.1,random_state = 33)
models_results['sgdc']=eval_model(name='SGDC', model=sgdc, X_train= X_sampled, y_train=y_sampled, X_test=X_test,y_test=y_test, treshold=0.5)

#from sklearn.tree import DecisionTreeClassifier
#dtc = DecisionTreeClassifier(max_depth = 10, random_state = 42)

# show all models in one picture to choose one 
# 
for model_name, result   in models_results.items():
    print(model_name, result)
    for stat,value in result.items():
        print(stat,value)
dfz=pd.DataFrame.from_dict(models_results,orient='index')
dfz.head(4) 

import sklearn.metrics as metrics
# calculate the fpr and tpr for all thresholds of the classification
probs = model.predict_proba(X_test)
preds = probs[:,1]
fpr, tpr, threshold = metrics.roc_curve(y_test, preds)
roc_auc = metrics.auc(fpr, tpr)

# method I: plt
import matplotlib.pyplot as plt
plt.title('Receiver Operating Characteristic')
plt.plot(fpr, tpr, 'b', label = 'AUC = %0.2f' % roc_auc)
plt.legend(loc = 'lower right')
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0, 1])
plt.ylim([0, 1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()

# most important features
model = LogisticRegression(solver='liblinear', penalty='l1')
# solver='lbfgs'
model.fit( X_train,y_train )
y_pred=model.predict(X_test)

feature_importances = pd.DataFrame(model.coef_[0],
                                   index = col2use,
                                    columns=['importance']).sort_values('importance',
                                                                        ascending=False)<jupyter_output><empty_output><jupyter_text>Final results<jupyter_code>feature_importances.head(15)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14431088424172567, 'fraction_numerical': 0.02460069158570723, 'mean_word_length': 4.774101540216772, 'pattern_counts': {'":': 1, '<': 64, '<?xml version=': 0, '>': 71, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '55593', 'license': 'Apache-2.0', 'n_tokens_mistral': 10879, 'n_tokens_neox': 10146, 'n_words': 2369, 'path': '_notebooks/Diabetic-readmisson-analysis.ipynb', 'repo_name': 'v2br/fastpages', 'size': '30372'}