<jupyter_start><jupyter_text>Comparison of the Machine Learning models performances for diabetes patientsMini-project to play with the classification methods: SVM, KNN and Random Forests. The goal is to understand the behavior of the classification methods and their applicability in different setups. **Context:**This dataset is originally from the National Institute of Diabetes and Digestive and Kidney Diseases. The objective of the dataset is to diagnostically predict whether or not a patient has diabetes, based on certain diagnostic measurements included in the dataset. Several constraints were placed on the selection of these instances from a larger database. In particular, all patients here are females at least 21 years old of Pima Indian heritage.**Content:**The datasets consists of several medical predictor variables and one target variable, Outcome. Predictor variables includes the number of pregnancies the patient has had, their BMI, insulin level, age, and so on.**The dataset is composed of:**- Pregnancies: Number of time pregnant- Glucose: Plasma glucose concentration a 2 hours in an oral glucose tolerance test- Blood Pressure: Diastolic blood pressure (mm Hg)- SkinThickness: Triceps skinfold thickness (mm)- Insulin: 2-Hour serum insulin (mu U/ml)- BMI: Body mass index (weight in kg/(height in m)^2)- Diabetes pedigree function: Diabetes pedigree function- Age: Age(years)- Outcome: 0 or 1  Import packages and data<jupyter_code>import pandas as pd
import numpy as np
import math
from datetime import datetime
import time 

from sklearn.datasets import load_digits

import seaborn as sns

import matplotlib.pyplot as plt

import missingno as msno

import struct

from sklearn import metrics
from sklearn import svm
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import confusion_matrix
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score,confusion_matrix,precision_score,recall_score,f1_score,roc_auc_score,roc_curve,make_scorer
from sklearn.model_selection import train_test_split,cross_val_score,RandomizedSearchCV
%matplotlib inline
from sklearn.inspection import permutation_importance
df_diabetes = pd.read_csv('diabetes.csv')
df_diabetes.head()
df_diabetes.shape
df_diabetes.dtypes
df_diabetes.describe()<jupyter_output><empty_output><jupyter_text>As you can see above some of the variables (Glucose,BloodPressure,SkinThickness,Insulin,BMI) have 0 as minimum value - which is impossible.Therefore we evaluate them as missing values and replace them by NaN and will later fill them by the median.<jupyter_code># Replacing 0 values by NaN
df_diabetes[['Glucose','BloodPressure','SkinThickness','Insulin','BMI']] = df_diabetes[['Glucose','BloodPressure','SkinThickness','Insulin','BMI']].replace(0,np.NaN)
# Check for missing values
msno.matrix(df_diabetes)
# Filling NaN values 
df_diabetes['Glucose'].fillna(df_diabetes['Glucose'].median(), inplace = True)
df_diabetes['BloodPressure'].fillna(df_diabetes['BloodPressure'].median(), inplace = True)
df_diabetes['SkinThickness'].fillna(df_diabetes['SkinThickness'].median(), inplace = True)
df_diabetes['Insulin'].fillna(df_diabetes['Insulin'].median(), inplace = True)
df_diabetes['BMI'].fillna(df_diabetes['BMI'].mean(), inplace = True)
# Check the features
print("Features: ", df_diabetes.columns)

# check the target outcome
df_diabetes.groupby('Outcome').size()<jupyter_output>Features:  Index(['Pregnancies', 'Glucose', 'BloodPressure', 'SkinThickness', 'Insulin',
       'BMI', 'DiabetesPedigreeFunction', 'Age', 'Outcome'],
      dtype='object')<jupyter_text>Data visualization<jupyter_code>df_diabetes.hist(figsize=(10,8))
fig, axarr = plt.subplots(3, 2, figsize=(12, 8))

sns.distplot(df_diabetes['Pregnancies'],ax=axarr[0][0])
sns.distplot(df_diabetes['Glucose'],ax=axarr[0][1])
sns.distplot(df_diabetes['BloodPressure'],ax=axarr[1][0])
sns.distplot(df_diabetes['SkinThickness'],ax=axarr[1][1])
sns.distplot(df_diabetes['Insulin'],ax=axarr[2][0])
sns.distplot(df_diabetes['BMI'],ax=axarr[2][1])

plt.subplots_adjust(hspace=1)
df_diabetes.plot(kind= 'box' , subplots=True, layout=(3,3), sharex=False, sharey=False, figsize=(10,8))
#Look for outliers in the df_diabetes
fig, axarr = plt.subplots(4, 2, figsize=(12, 20))
sns.boxplot(df_diabetes['Outcome'], df_diabetes['Pregnancies'],ax=axarr[0][0])
sns.boxplot(df_diabetes['Outcome'], df_diabetes['Glucose'],ax=axarr[0][1])
sns.boxplot(df_diabetes['Outcome'], df_diabetes['BloodPressure'],ax=axarr[1][0])
sns.boxplot(df_diabetes['Outcome'], df_diabetes['SkinThickness'],ax=axarr[1][1])
sns.boxplot(df_diabetes['Outcome'], df_diabetes['Insulin'],ax=axarr[2][0])
sns.boxplot(df_diabetes['Outcome'], df_diabetes['BMI'],ax=axarr[2][1])
sns.boxplot(df_diabetes['Outcome'], df_diabetes['Age'],ax=axarr[3][0])
sns.boxplot(df_diabetes['Outcome'], df_diabetes['DiabetesPedigreeFunction'],ax=axarr[3][1])
X = df_diabetes.drop(columns='Outcome',axis=1)
#X = df_diabetes[['Pregnancies','BMI','DiabetesPedigreeFunction']]
Y = df_diabetes['Outcome']

X_train_diabetes, X_test_diabetes, Y_train_diabetes, Y_test_diabetes = train_test_split(X,Y, test_size=0.2,random_state=42) # 80% training and 20% test

#Standard scaling
sc = StandardScaler()
X_train = sc.fit_transform(X_train_diabetes)
X_test = sc.transform(X_test_diabetes)
# Getting correlations of each features 
corrmat = df_diabetes.corr()
top_corr_features = corrmat.index
plt.figure(figsize = (10,8))

g = sns.heatmap(df_diabetes[top_corr_features].corr(),annot = True,cmap = "YlGnBu")<jupyter_output><empty_output><jupyter_text>1. SVM<jupyter_code>svm_t_start = time.time() 

#Create a svm Classifier
svm_diabetes = svm.SVC(kernel='linear') # Linear Kernel

svm_diabetes.fit(X_train_diabetes, Y_train_diabetes)

y_pred_diabetes = svm_diabetes.predict(X_test_diabetes)

print(f'Training took (s): {time.time()-svm_t_start}')
# Model Accuracy: how often is the classifier correct?
svm_accuracy = metrics.accuracy_score(Y_test_diabetes, y_pred_diabetes)
print("Accuracy:",svm_accuracy)

#positive predictive value
svm_precision = metrics.precision_score(Y_test_diabetes, y_pred_diabetes)
print("Precision:",svm_precision)

#sensitivity
svm_recall = metrics.recall_score(Y_test_diabetes, y_pred_diabetes)
print("Recall:",svm_recall)
print (metrics.classification_report(Y_test_diabetes, y_pred_diabetes))
#Apply k-fold validation here
svm_cv_diabetes = cross_val_score(svm_diabetes,X_train_diabetes,Y_train_diabetes,cv=10)
svm_cv_diabetes

svm_cv_mean = np.mean(svm_cv_diabetes)

print(svm_cv_diabetes)
print('svm_cv_diabetes mean:{}'.format(svm_cv_mean))<jupyter_output>[0.72580645 0.74193548 0.85483871 0.75806452 0.73770492 0.81967213
 0.72131148 0.73770492 0.78688525 0.78688525]
svm_cv_diabetes mean:0.7670809095716552<jupyter_text>C ParameterThe C parameter tells the SVM optimization how much you want to avoid misclassifying each training example. For large values of C, the optimization will choose a smaller-margin hyperplane if that hyperplane does a better job of getting all the training points classified correctly. Conversely, a very small value of C will cause the optimizer to look for a larger-margin separating hyperplane, even if that hyperplane misclassifies more points. For very tiny values of C, you should get misclassified examples, often even if your training data is linearly separable.<jupyter_code>def cparameter(i):
    model = svm.SVC(kernel='linear',C=i)
    model.fit(X_train_diabetes, Y_train_diabetes)
    y_pred = model.predict(X_test_diabetes)
    matrix = confusion_matrix(Y_test_diabetes, y_pred)
    print(metrics.accuracy_score(Y_test_diabetes, y_pred))
    print (matrix)
l = [0.1,1,10]
for i in l:
    print("for the value of c = ", i)
    cparameter(i)<jupyter_output>for the value of c =  0.1
0.7597402597402597
[[82 17]
 [20 35]]
for the value of c =  1
0.7597402597402597
[[82 17]
 [20 35]]
for the value of c =  10
0.7662337662337663
[[84 15]
 [21 34]]<jupyter_text>GridSearchCV<jupyter_code>param_grid = [
  {'C': [0.1,1,10,100], 'gamma': [0.01,0.001], 'kernel': ['rbf','sigmoid']},
 ]
grid=GridSearchCV(svm_diabetes,param_grid=param_grid, cv=10, n_jobs=-1)
grid.fit(X_train_diabetes,Y_train_diabetes)
y_pred=grid.predict(X_test_diabetes)
grid.best_score_
grid.best_params_<jupyter_output><empty_output><jupyter_text>AUC<jupyter_code>fpr, tpr, thresholds = metrics.roc_curve(Y_test_diabetes, y_pred_diabetes)
svm_auc = metrics.auc(fpr, tpr)<jupyter_output><empty_output><jupyter_text>Feature Importance<jupyter_code>pd.Series(abs(svm_diabetes.coef_[0]), index=X_train_diabetes.columns).nlargest(10).plot(kind='barh')
svm_feature_imp = abs(svm_diabetes.coef_[0])<jupyter_output><empty_output><jupyter_text>2. KNN<jupyter_code>neighbors = np.arange(1,18)
train_accuracy =np.empty(len(neighbors))
test_accuracy = np.empty(len(neighbors))

for i,k in enumerate(neighbors):
   
    knn = KNeighborsClassifier(n_neighbors=k)

    knn.fit(X_train_diabetes, Y_train_diabetes)
    train_accuracy[i] = knn.score(X_train_diabetes, Y_train_diabetes)
    test_accuracy[i] = knn.score(X_test_diabetes, Y_test_diabetes) 
plt.figure(figsize=(12,7))
plt.title('k for k-NN')
plt.plot(neighbors, test_accuracy, label='Testing Accuracy')
plt.plot(neighbors, train_accuracy, label='Training accuracy')

plt.legend()
plt.xlabel('Number of neighbors')
plt.ylabel('Accuracy')
plt.show()<jupyter_output><empty_output><jupyter_text>Observation: We can observe above that we get maximum testing accuracy for k=12<jupyter_code>knn_t_start = time.time() 

# Create KNN Classifier
knn_diabetes = KNeighborsClassifier(n_neighbors=14)

knn_diabetes.fit(X_train_diabetes, Y_train_diabetes)

knn_y_pred_diabetes = knn_diabetes.predict(X_test_diabetes)

print(f'Training took (s): {time.time()-knn_t_start}')
# Model Accuracy: how often is the classifier correct?
knn_accuracy = metrics.accuracy_score(Y_test_diabetes, knn_y_pred_diabetes)
print("Accuracy:",knn_accuracy)

# Model Precision: what percentage of positive tuples are labeled as such?
knn_precision = metrics.precision_score(Y_test_diabetes, knn_y_pred_diabetes)
print("Precision:",knn_precision)

# Model Recall: what percentage of positive tuples are labelled as such?
knn_recall = metrics.recall_score(Y_test_diabetes, knn_y_pred_diabetes)
print("Recall:",knn_recall)
#Apply k-fold validation here
knn_cv_diabetes = cross_val_score(knn_diabetes,X_train_diabetes,Y_train_diabetes,cv=5)

knn_cv_mean = np.mean(knn_cv_diabetes)
print(knn_cv_diabetes)
print('knn_cv_scores mean:{}'.format(knn_cv_mean))<jupyter_output>[0.70731707 0.7398374  0.74796748 0.73170732 0.7704918 ]
knn_cv_scores mean:0.7394642143142743<jupyter_text>GridSearchCV<jupyter_code>param_grid_knn = [
  {'n_neighbors': [10,12,14], 'weights': ['uniform','distance'], 'metric': ['euclidean','manhattan']},
 ]
grid_knn=GridSearchCV(knn_diabetes,param_grid=param_grid_knn, cv=10, n_jobs=-1)
grid_knn.fit(X_train_diabetes,Y_train_diabetes)
y_pred=grid.predict(X_test_diabetes)
grid_knn.best_score_
grid_knn.best_params_<jupyter_output><empty_output><jupyter_text>Feature Importance<jupyter_code>results = permutation_importance(knn_diabetes, X_train_diabetes, Y_train_diabetes, scoring='accuracy')
knn_feature_imp = results.importances_mean
knn_feature_imp
index = np.array(X_train_diabetes.columns)
df_knn_imp_features = pd.DataFrame({'score':np.array(knn_feature_imp), 'features':np.array(X_train_diabetes.columns)}, index=index)

df_knn_imp_features.sort_values(by="score", ascending = False).plot.bar(rot=90)<jupyter_output><empty_output><jupyter_text>AUC<jupyter_code>fpr, tpr, thresholds = metrics.roc_curve(Y_test_diabetes, knn_y_pred_diabetes)
knn_auc = metrics.auc(fpr, tpr)<jupyter_output><empty_output><jupyter_text>3. Random Forests<jupyter_code># Instantiating random forest classifier

clf = RandomForestClassifier(oob_score = True,n_jobs = -1,random_state = 100)
clf
# Creating Predictor Matrix
X = df_diabetes.drop('Outcome',axis = 1)

# Target variable
y = df_diabetes['Outcome']
# Splitting the matrices into random train & test subsets where test data contains 25% data and rest considered as training data

X_train,X_test,y_train,y_test = train_test_split(X,y,test_size = 0.2,random_state = 200)
# Performing K-fold cross validation with 5 folds 

scores = cross_val_score(clf,X_train,y_train,cv = 5,scoring = "f1_macro")
scores.mean()
# Building a forest of trees from training set
knn_t_start = time.time() 
clf.fit(X_train,y_train)
print(f'Training took (s): {time.time()-knn_t_start}')
#Predicting on classifier created

train_pred = clf.predict(X_train)
test_pred = clf.predict(X_test)
# Finding F1 score of training and testing sets 

print("The training F1 score is: ",f1_score(train_pred,y_train))
print("The testing F1 score is :",f1_score(test_pred,y_test))<jupyter_output>The training F1 score is:  1.0
The testing F1 score is : 0.68<jupyter_text>Model is overfitting as testing F1 score is lower than training. Hyperparameter tuning needs to be done and looking for high F1 score that is why scorer variable is defined.<jupyter_code>#  Tuning hyperparameters

parameters = {
             "max_depth":[2,3,4],
             "n_estimators":[100,104,106],
             "min_samples_split":[3,4,5],
             "min_samples_leaf":[4,8,9]
             }

scorer = make_scorer(f1_score)
# Using Randomized Search CV to find best optimal hyperparameter that best describe a classifier

clf1 = RandomizedSearchCV(clf,parameters,scoring = scorer)

# Fitting the model

clf1.fit(X_train,y_train)

# Getting best estimator having high score

best_clf_random = clf1.best_estimator_
best_clf_random

# Again, finding cross validation score

scores = cross_val_score(best_clf_random,X_train,y_train,cv = 5,scoring = "f1_macro")
rf_cv_mean = scores.mean()<jupyter_output><empty_output><jupyter_text>As you can see cross validation score has decreased as compared to earlier score. It should increase and for that you have to try changing hyperparameter values so that better cross validation score can be achieved.<jupyter_code># Fitting the best estimator

best_clf_random.fit(X_train,y_train)
# Getting first estimator

best_clf_random.estimators_[0]
# Predicting on best estimator

train_pred = best_clf_random.predict(X_train)
test_pred = best_clf_random.predict(X_test)
# Finding the F1 score of training & testing sets

print("The training F1 score is: ",f1_score(train_pred,y_train))
print("The testing F1 score is :",f1_score(test_pred,y_test))<jupyter_output>The training F1 score is:  0.6770833333333333
The testing F1 score is : 0.6436781609195402<jupyter_text>No the testing F1 score is higher than training score.<jupyter_code># Getting accuracy score 

rf_accuracy = accuracy_score(y_test,test_pred)
# Computing ROC AUC from prediction scores

rf_auc = roc_auc_score(y_test,best_clf_random.predict_proba(X_test)[:,1])
# Plotting ROC curve

fpr,tpr,thresholds = roc_curve(y_test,best_clf_random.predict_proba(X_test)[:,1])

plt.plot([0,1],[0,1],'k--')
plt.plot(fpr,tpr)
plt.xlabel('fpr')
plt.ylabel('tpr')
plt.show()
# Computing confusion matrix

pd.crosstab(y_test,test_pred,rownames = ['True'],colnames = ['Predicted'],margins = True)
# Plotting confusion matrix
cnf_matrix = confusion_matrix(y_test,test_pred)
p = sns.heatmap(pd.DataFrame(cnf_matrix),annot = True,cmap = "YlGnBu",fmt = 'g')
plt.title("Confusion Matrix",y = 1.1)
plt.xlabel('Predicted Label')
plt.ylabel('Actual Label')
# Computing the precision

rf_precision = precision_score(y_test,test_pred)
# Computing the recall

rf_recall = recall_score(y_test,test_pred)
# Getting feature importances
rf_features = pd.Series(best_clf_random.feature_importances_,index = X.columns)
rf_features.sort_values(ascending = False)
# Plotting feature importances 
rf_features.sort_values(ascending = False).plot(kind = "bar")
svm_f1 = 2 * (svm_precision * svm_recall) / (svm_precision + svm_recall)
knn_f1 = 2 * (knn_precision * knn_recall) / (knn_precision + knn_recall)
rf_f1 = 2 * (rf_precision * rf_recall) / (rf_precision + rf_recall)
plt.figure(figsize=(13,7))
score_name= ['Accuracy','CV Accuracy', 'AUC', 'F1 Score', 'Recall', 'Precision'] 
plt.plot(score_name, [svm_accuracy,svm_cv_mean,svm_f1,svm_auc,svm_recall,svm_precision], 'o', label='SVM')
plt.plot(score_name, [knn_accuracy,knn_cv_mean,knn_auc,knn_f1,knn_recall,knn_precision], 'o', label='KNN')
plt.plot(score_name, [rf_accuracy, rf_cv_mean, rf_auc, rf_f1, rf_recall, rf_precision], 'o', label='Random forests')
plt.legend()
plt.xlabel('Scores')
plt.ylabel('Percentage')
plt.title('Comparison of Models Scores of SVM, kNN, Random Forests')
plt.show()<jupyter_output><empty_output><jupyter_text>Feature importances Comparison<jupyter_code>svm = svm_feature_imp
knn = knn_feature_imp

random_forests = best_clf_random.feature_importances_
index = np.array(X_train_diabetes.columns)
df = pd.DataFrame({'SVM': svm, 'kNN': knn, 'Random Forests': random_forests}, index=index)
ax = df.plot.bar(figsize=(13,7),rot=90, title='Comparison of Feature Importance of SVM, kNN, Random Forests')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11373278316964026, 'fraction_numerical': 0.028709246236996572, 'mean_word_length': 7.775114737378888, 'pattern_counts': {'":': 4, '<': 67, '<?xml version=': 0, '>': 67, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '488338', 'license': 'MIT', 'n_tokens_mistral': 6758, 'n_tokens_neox': 6122, 'n_words': 1305, 'path': 'notebook.ipynb', 'repo_name': 'yuliianikolaenko/diabetes-ml-models', 'size': '17207'}