<jupyter_start><jupyter_text>PIMA Indians Diabetes Background**Diabetes**, is a group of metabolic disorders in which there are high blood sugar levels over a prolonged period.  Symptoms of high blood sugar include frequent urination, increased thirst, and increased hunger.  If left untreated, diabetes can cause many complications.  Acute complications can include diabetic ketoacidosis, hyperosmolar hyperglycemic state, or death.  Serious long-term complications include cardiovascular disease, stroke, chronic kidney disease, foot ulcers, and damage to the eyes.This **dataset** is originally from the National Institute of Diabetes and Digestive and Kidney Diseases. The objective of the dataset is to diagnostically predict whether or not a patient has diabetes, based on certain diagnostic measurements included in the dataset. Several constraints were placed on the selection of these instances from a larger database. In particular, all patients here are females at least 21 years old of Pima Indian heritage. ObjectiveWe will try to build a machine learning model to accurately predict whether or not the patients in the dataset have diabetes or not? DataThe datasets consists of several medical predictor variables and one target variable, Outcome. Predictor variables includes the number of pregnancies the patient has had, their BMI, insulin level, age, and so on.1. **Pregnancies**: Number of times pregnant2. **Glucose**: Plasma glucose concentration a 2 hours in an oral glucose tolerance test3. **BloodPressure**: Diastolic blood pressure (mm Hg)4. **SkinThickness**: Triceps skin fold thickness (mm)5. **Insulin**: 2-Hour serum insulin (mu U/ml)6. **BMI**: Body mass index (weight in kg/(height in m)^2)7. **DiabetesPedigreeFunction**: Diabetes pedigree function8. **Age**: Age (years)9. **Outcome**: Class variable (0 or 1)<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
df = pd.read_csv('data/diabetes.csv')
print(df.shape)
df.head(10)<jupyter_output>(768, 9)<jupyter_text>0. Load data 1. Identifying and correcting Data types 2. Data Visualisation & Feature Selection 3. Fixing Missing Values - Imputer 4. Data Scaling- StandScaler / MinMaxScaler 5. Modelling  1. Identifying and correcting Data types<jupyter_code>df.dtypes<jupyter_output><empty_output><jupyter_text>2. Data Visualisation & Feature Selection<jupyter_code>df[df.columns[:8]].hist(figsize=(10,10));<jupyter_output><empty_output><jupyter_text>Dist PlotDist Plot helps us to flexibly plot a univariate distribution of observations.<jupyter_code>fig, ax = plt.subplots(4,2, figsize=(16,16))
sns.distplot(df.Age, bins = 20, ax=ax[0,0]) 
sns.distplot(df.Pregnancies, bins = 20, ax=ax[0,1]) 
sns.distplot(df.Glucose, bins = 20, ax=ax[1,0]) 
sns.distplot(df.BloodPressure, bins = 20, ax=ax[1,1]) 
sns.distplot(df.SkinThickness, bins = 20, ax=ax[2,0])
sns.distplot(df.Insulin, bins = 20, ax=ax[2,1])
sns.distplot(df.DiabetesPedigreeFunction, bins = 20, ax=ax[3,0]) 
sns.distplot(df.BMI, bins = 20, ax=ax[3,1])<jupyter_output><empty_output><jupyter_text>output shows that there are many values for the coloumns that are zero.<jupyter_code># Lets check the figures as well
for c in df.columns:
    if c not in ['Pregnancies', 'Outcome']:
        print ('0 values in', c, ':', sum(df[c]==0))
sns.regplot(x='SkinThickness', y= 'Insulin', data=df)<jupyter_output><empty_output><jupyter_text>Violin Plots A violin plot is a method of plotting numeric data. It is similar to box plot with a rotated kernel density plot on each side. Violin plots are similar to box plots, except that they also show the probability density of the data at different values (in the simplest case this could be a histogram).  A violin plot is more informative than a plain box plot. In fact while a box plot only shows summary statistics such as mean/median and interquartile ranges, the violin plot shows the full distribution of the data. The difference is particularly useful when the data distribution is multimodal (more than one peak). In this case a violin plot clearly shows the presence of different peaks, their position and relative amplitude. This information could not be represented with a simple box plot which only reports summary statistics. The inner part of a violin plot usually shows the mean (or median) and the interquartile range.<jupyter_code>fig,ax = plt.subplots(nrows=4, ncols=2, figsize=(18,18))
plt.suptitle('Violin Plots',fontsize=24)
sns.violinplot(x="Pregnancies", data=df,ax=ax[0,0],palette='Set3')
sns.violinplot(x="Glucose", data=df,ax=ax[0,1],palette='Set3')
sns.violinplot (x ='BloodPressure', data=df, ax=ax[1,0], palette='Set3')
sns.violinplot(x='SkinThickness', data=df, ax=ax[1,1],palette='Set3')
sns.violinplot(x='Insulin', data=df, ax=ax[2,0], palette='Set3')
sns.violinplot(x='BMI', data=df, ax=ax[2,1],palette='Set3')
sns.violinplot(x='DiabetesPedigreeFunction', data=df, ax=ax[3,0],palette='Set3')
sns.violinplot(x='Age', data=df, ax=ax[3,1],palette='Set3')
plt.show()<jupyter_output><empty_output><jupyter_text>Correlation between features Variables within a dataset can be related for lots of reasons. It can be useful in data analysis and modeling to better understand the relationships between variables. The statistical relationship between two variables is referred to as their correlation. A correlation could be positive, meaning both variables move in the same direction, or negative, meaning that when one variable’s value increases, the other variables’ values decrease. Correlation can also be neural or zero, meaning that the variables are unrelated.<jupyter_code>corr=df.corr()
corr
sns.set(font_scale=1.15)
plt.figure(figsize=(14, 10))

sns.heatmap(corr, vmax=.8, linewidths=0.01,
            square=True,annot=True,cmap='coolwarm',linecolor="black")
plt.title('Correlation between features');<jupyter_output><empty_output><jupyter_text>3. Fixing Missing Values - Imputer<jupyter_code>df.isnull().sum()
from sklearn.impute import SimpleImputer
imputer = SimpleImputer(missing_values=0, strategy='median')
df[df.columns[1:8]] = pd.DataFrame(imputer.fit_transform(df[df.columns[1:8]]))
# Lets check the figures again
for c in df.columns:
    if c not in ['Pregnancies', 'Outcome']:
        print ('0 values in', c, ':', sum(df[c]==0))<jupyter_output>0 values in Glucose : 0
0 values in BloodPressure : 0
0 values in SkinThickness : 0
0 values in Insulin : 0
0 values in BMI : 0
0 values in DiabetesPedigreeFunction : 0
0 values in Age : 0<jupyter_text>4. Data Scaling- StandScaler / MinMaxScaler<jupyter_code>df.describe()
from sklearn.preprocessing import RobustScaler
scaler = RobustScaler()
scaler.fit(df[df.columns[0:8]])
df[df.columns[0:8]] = scaler.transform(df[df.columns[0:8]])
df.describe()<jupyter_output><empty_output><jupyter_text>5. Modelling  Dataset Preparation<jupyter_code>from sklearn.metrics import accuracy_score
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import GradientBoostingClassifier

X = df.iloc[:, :-1]
y = df.iloc[:, -1]


X_train, X_test, y_train, y_test = train_test_split(X, y, 
                                                    test_size=0.25, 
                                                    random_state=0)<jupyter_output><empty_output><jupyter_text>Logistic Regression<jupyter_code>from sklearn.linear_model import LogisticRegression

#Model
LR = LogisticRegression(solver="sag", max_iter=1000)

#fiting the model
LR.fit(X_train, y_train)

#prediction
y_pred = LR.predict(X_test)

#Accuracy
print("Testing Accuracy ", LR.score(X_test, y_test)*100)
#Plot the confusion matrix
sns.set(font_scale=1.5)
cm = confusion_matrix(y_pred, y_test)
sns.heatmap(cm, annot=True, fmt='g')
plt.show()
## Read more here: 
#1. https://towardsdatascience.com/dont-sweat-the-solver-stuff-aea7cddc3451
#2. https://stackoverflow.com/questions/38640109/logistic-regression-python-solvers-defintions
from sklearn.model_selection import GridSearchCV
solver_list = ['liblinear', 'newton-cg', 'lbfgs', 'sag', 'saga']
params = dict(solver=solver_list)
log_reg = LogisticRegression(C=1, random_state=34, max_iter=3000)
clf = GridSearchCV(log_reg, params, cv=5)
clf.fit(X_train, y_train)
scores = clf.cv_results_['mean_test_score']

for score, solver in zip(scores, solver_list):
    print(f"  {solver} {score:.3f}" )<jupyter_output>liblinear 0.767
  newton-cg 0.767
  lbfgs 0.767
  sag 0.771
  saga 0.767<jupyter_text>SVM<jupyter_code>from sklearn.svm import SVC

# creating the model
model = SVC(gamma="auto", 
            kernel='rbf', #radial basis function
            class_weight="balanced")

# feeding the training data into the model
model.fit(X_train, y_train)

# predicting the test set results
y_pred = model.predict(X_test)

# Calculating the accuracies
print("Training accuracy :", model.score(X_train, y_train))
print("Testing accuracy :", model.score(X_test, y_test))

# classification report
print(classification_report(y_test, y_pred))

# confusion matrix 
print(confusion_matrix(y_test, y_pred))<jupyter_output>Training accuracy : 0.8263888888888888
Testing accuracy : 0.7864583333333334
              precision    recall  f1-score   support

           0       0.87      0.81      0.84       130
           1       0.65      0.74      0.69        62

    accuracy                           0.79       192
   macro avg       0.76      0.77      0.76       192
weighted avg       0.80      0.79      0.79       192

[[105  25]
 [ 16  46]]<jupyter_text>Decision Tree<jupyter_code>#Model
DT = DecisionTreeClassifier()

#fiting the model
DT.fit(X_train, y_train)

#prediction
y_pred = DT.predict(X_test)

#Accuracy
print("Testing Accuracy ", DT.score(X_test, y_test)*100)

#Plot the confusion matrix
sns.set(font_scale=1.5)
cm = confusion_matrix(y_pred, y_test)
sns.heatmap(cm, annot=True, fmt='g')
plt.show()<jupyter_output>Testing Accuracy  76.5625<jupyter_text>Hyperparameter Tuning<jupyter_code>from sklearn.model_selection import GridSearchCV

# Decision Tree hyperparameter tuning.
DTC = DecisionTreeClassifier()

# Define parameter grid.
dt_param_grid = {"max_depth": [4, 6, 8, 10],
                 "max_features": ["sqrt", "log2", 2, 5, 8],
                 "min_samples_split": [3, 4, 5, 10],
                 "min_samples_leaf": [3, 4, 5, 10],
                 "criterion": ["gini", "entropy"],
                 "class_weight": [None, 'balanced']
                }

# Search grid and store best estimator.
gsDTC = GridSearchCV(DTC,param_grid = dt_param_grid, cv=5, 
                     scoring="f1", n_jobs= 4)
gsDTC.fit(X_train,y_train)

#Accuracy
print("Testing Accuracy ", gsDTC.score(X_test, y_test)*100)

DTC_best = gsDTC.best_estimator_

# Print best score.
print('Best score: {}'.format(gsDTC.best_score_))
print('Best parameters: {}'.format(gsDTC.best_params_))<jupyter_output>Testing Accuracy  67.08860759493672
Best score: 0.6885254234816345
Best parameters: {'class_weight': 'balanced', 'criterion': 'gini', 'max_depth': 4, 'max_features': 5, 'min_samples_leaf': 3, 'min_samples_split': 4}<jupyter_text>Random Forest<jupyter_code>from sklearn.ensemble import RandomForestClassifier

# Random Forest hyperparameter tuning. 
RFC = RandomForestClassifier()

# Define parameter grid.
rf_param_grid = {"max_depth": [4, 6, 8, 10],
                 "max_features": ["sqrt", "log2", 2, 5, 8],
                 "min_samples_split": [3, 4, 5, 8],
                 "min_samples_leaf": [3, 4, 5, 8],
                 "bootstrap": [False, True],
                 "n_estimators" :[100],
                 "criterion": ["gini", "entropy"],
                 "class_weight": [None, 'balanced']
                }
                
# Search grid and store best estimator.
gsRFC = GridSearchCV(RFC,param_grid = rf_param_grid, cv=4, scoring="f1", n_jobs= 4)
gsRFC.fit(X_train,y_train)
RFC_best = gsRFC.best_estimator_

# Print best score.
print('Best score: {}'.format(gsRFC.best_score_))
print('Best parameters: {}'.format(gsRFC.best_params_))
#Look at feature importances for best model.
RFC_best.fit(X_train,y_train)
features_best = pd.DataFrame()
features_best['feature'] = X_train.columns
features_best['importance'] = RFC_best.feature_importances_
features_best.sort_values(by=['importance'], ascending=True, inplace=True)
features_best.set_index('feature', inplace=True)

#Plot.
print("\nFeature Importances for Best Model:")
ax = features_best.plot(kind='barh', figsize=(20, 14), fontsize=20)
ax.set_ylabel("Features", fontname="Arial", fontsize=24)
# Print Classification Report for both training and test data.
test_predictions = RFC_best.predict(X_test)
train_predictions = RFC_best.predict(X_train)
print('\n***Training Data***')
print('\nClassification Report:')
print(classification_report(y_train, train_predictions))
print('\n***Test Data***')
print('\nClassification Report:')
print(classification_report(y_test, test_predictions))<jupyter_output><empty_output><jupyter_text>Adaboost<jupyter_code>from sklearn.ensemble import AdaBoostClassifier


model = AdaBoostClassifier(DecisionTreeClassifier(max_depth=None),n_estimators=100, random_state=42)

#fiting the model
model.fit(X_train, y_train)

#prediction
y_pred = model.predict(X_test)

#Accuracy
print("Accuracy ", model.score(X_test, y_test)*100)

#Plot the confusion matrix
sns.set(font_scale=1.5)
cm = confusion_matrix(y_pred, y_test)
sns.heatmap(cm, annot=True, fmt='g')
plt.show()<jupyter_output>Accuracy  76.5625<jupyter_text>Gradient Boosting Gradient boosting is a machine learning technique for regression and classification problems, which produces a prediction model in the form of an ensemble of weak prediction models, typically decision trees. It builds the model in a stage-wise fashion like other boosting methods do, and it generalizes them by allowing optimization of an arbitrary differentiable loss function.<jupyter_code>#Model
model = GradientBoostingClassifier()

#fiting the model
model.fit(X_train, y_train)

#prediction
y_pred = model.predict(X_test)

#Accuracy
print("Accuracy ", model.score(X_test, y_test)*100)

#Plot the confusion matrix
sns.set(font_scale=1.5)
cm = confusion_matrix(y_pred, y_test)
sns.heatmap(cm, annot=True, fmt='g')
plt.show()<jupyter_output>Accuracy  80.72916666666666<jupyter_text>Stacking  First level modelDefining the first level models for stacking.  We are using three models: - KNN-Classifier, - Random Forest Classifier- XGBoost Classifier.<jupyter_code># !pip install vecstack
# Also refer to: https://github.com/vecxoz/vecstack
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier
from xgboost import XGBClassifier
from vecstack import stacking
models = [
    KNeighborsClassifier(n_neighbors=5,
                        n_jobs=-1),
        
    RandomForestClassifier(random_state=0, n_jobs=-1, 
                           n_estimators=100, max_depth=3),
        
    XGBClassifier(random_state=0, n_jobs=-1, learning_rate=0.1, 
                  n_estimators=100, max_depth=3)
]
S_train, S_test = stacking(models, X_train, y_train, X_test, 
                           regression=False, mode='oof_pred_bag', 
                           needs_proba=False, save_dir=None, 
                           metric=accuracy_score, n_folds=4, 
                           stratified=True, shuffle=True, 
                           random_state=0,    
                           verbose=2)<jupyter_output><empty_output><jupyter_text>Second level model<jupyter_code>model = XGBClassifier(random_state=0, n_jobs=-1, learning_rate=0.1, 
                      n_estimators=100, max_depth=3)
    
model = model.fit(S_train, y_train)
y_pred = model.predict(S_test)
print('Final prediction score: [%.8f]' % accuracy_score(y_test, y_pred))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10908519333542911, 'fraction_numerical': 0.02986482238289846, 'mean_word_length': 4.668567355666429, 'pattern_counts': {'":': 13, '<': 69, '<?xml version=': 0, '>': 69, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 3, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '472861', 'license': 'MIT', 'n_tokens_mistral': 5616, 'n_tokens_neox': 5142, 'n_words': 1421, 'path': 'src/week5/day2_pima-indians-diabetes.ipynb', 'repo_name': 'skyimager/intro-to-ml-with-python', 'size': '15905'}