<jupyter_start><jupyter_text>Grid Search Hyperparameter optimization This case study is all about using grid searches to identify the optimal parameters for a machine learning algorithm. To complere this case study, you'll use the Pima Indian diabetes dataset from Kaggle and KNN. Follow along with the preprocessing steps of this case study. Load the necessary packages<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()<jupyter_output><empty_output><jupyter_text>Load the diabetes data<jupyter_code>diabetes_data = pd.read_csv('diabetes.csv')
diabetes_data.head()<jupyter_output><empty_output><jupyter_text>** Start by reviewing the data info.**<jupyter_code>diabetes_data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 768 entries, 0 to 767
Data columns (total 9 columns):
 #   Column                    Non-Null Count  Dtype  
---  ------                    --------------  -----  
 0   Pregnancies               768 non-null    int64  
 1   Glucose                   768 non-null    int64  
 2   BloodPressure             768 non-null    int64  
 3   SkinThickness             768 non-null    int64  
 4   Insulin                   768 non-null    int64  
 5   BMI                       768 non-null    float64
 6   DiabetesPedigreeFunction  768 non-null    float64
 7   Age                       768 non-null    int64  
 8   Outcome                   768 non-null    int64  
dtypes: float64(2), int64(7)
memory usage: 54.1 KB<jupyter_text>** Apply the describe function to the data.**<jupyter_code>diabetes_data.describe()<jupyter_output><empty_output><jupyter_text>** Currently, the missing values in the dataset are represented as zeros. Replace the zero values in the following columns ['Glucose','BloodPressure','SkinThickness','Insulin','BMI'] with nan .**<jupyter_code>diabetes_data = diabetes_data.astype(float)
diabetes_data.loc[:,['Glucose','BloodPressure','SkinThickness','Insulin','BMI']].replace(0 , np.nan, inplace=True)<jupyter_output><empty_output><jupyter_text>** Plot histograms of each column. **<jupyter_code>diabetes_data.hist()
plt.gcf().set_size_inches(10,10)
plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>Replace the zeros with mean and median values.<jupyter_code>diabetes_data['Glucose'].fillna(diabetes_data['Glucose'].mean(), inplace = True)
diabetes_data['BloodPressure'].fillna(diabetes_data['BloodPressure'].mean(), inplace = True)
diabetes_data['SkinThickness'].fillna(diabetes_data['SkinThickness'].median(), inplace = True)
diabetes_data['Insulin'].fillna(diabetes_data['Insulin'].median(), inplace = True)
diabetes_data['BMI'].fillna(diabetes_data['BMI'].median(), inplace = True)
diabetes_data.head()<jupyter_output><empty_output><jupyter_text>** Plot histograms of each column after replacing nan. **<jupyter_code>diabetes_data.hist()
plt.gcf().set_size_inches(10,10)
plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>Plot the correlation matrix heatmap<jupyter_code>plt.figure(figsize=(12,10))
print('Correlation between various features')
p=sns.heatmap(diabetes_data.corr(), annot=True,cmap ='Blues')<jupyter_output>Correlation between various features<jupyter_text>** Using Sklearn, standarize the magnitude of the features by scaling the values. **<jupyter_code>from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split
X = diabetes_data.drop(['Outcome'], axis=1)
scaler = MinMaxScaler()
scaler.fit(X)
scaled_df = scaler.transform(X)<jupyter_output><empty_output><jupyter_text>** Define the `y` variable as the `Outcome` column.**<jupyter_code>y=diabetes_data.pop('Outcome')<jupyter_output><empty_output><jupyter_text>** Create a 70/30 train and test split. **<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=42)<jupyter_output><empty_output><jupyter_text>Using a range of neighbor values of 1-10, apply the KNearestNeighbor classifier to classify the the data.<jupyter_code>from sklearn.neighbors import KNeighborsClassifier


test_scores = []
train_scores = []
max_k = 10

for i in range(1,max_k):

    knn = KNeighborsClassifier(i)
    knn.fit(X_train,y_train)
    
    train_scores.append(knn.score(X_train,y_train))
    test_scores.append(knn.score(X_test,y_test))<jupyter_output><empty_output><jupyter_text>** Print the train and test scores for each iteration.**<jupyter_code>scores = pd.DataFrame({'Train_score':train_scores, 'Test_score':test_scores})
print(scores)<jupyter_output>Train_score  Test_score
0     1.000000    0.688312
1     0.841713    0.727273
2     0.843575    0.675325
3     0.811918    0.722944
4     0.802607    0.688312
5     0.795158    0.701299
6     0.800745    0.692641
7     0.789572    0.714286
8     0.789572    0.701299<jupyter_text>** Identify the number of neighbors between 1-15 that resulted in the max score in the training dataset. **<jupyter_code>best_train_knn = scores.Train_score.idxmax()+1
print(best_train_knn)<jupyter_output>1<jupyter_text>** Identify the number of neighbors between 1-15 that resulted in the max score in the testing dataset. **<jupyter_code>best_test_knn = scores.Test_score.idxmax()+1
print(best_test_knn)<jupyter_output>2<jupyter_text>Plot the train and test model performance by number of neighbors.<jupyter_code>plt.figure(figsize=(12,5))
p = sns.lineplot(range(1,max_k),train_scores,marker='*',label='Train Score')
p = sns.lineplot(range(1,max_k),test_scores,marker='o',label='Test Score')<jupyter_output><empty_output><jupyter_text>** Fit and score the best number of neighbors based on the plot. **<jupyter_code>#let's choose k=9 because it's odd and it looks like a good balance between variance ansd bias
k=9
knn = KNeighborsClassifier(9)
knn.fit(X_train,y_train)
    
train_scores.append(knn.score(X_train,y_train))
test_scores.append(knn.score(X_test,y_test))
from sklearn.metrics import confusion_matrix
y_pred = knn.predict(X_test)
pl = confusion_matrix(y_test,y_pred)
print(pl)<jupyter_output>[[115  36]
 [ 33  47]]<jupyter_text>** Plot the confusion matrix for the model fit above. **<jupyter_code>from sklearn.metrics import plot_confusion_matrix,  classification_report

# Plot non-normalized confusion matrix
titles_options = [("Confusion matrix, without normalization", None),
                  ("Normalized confusion matrix", 'true')]
for title, normalize in titles_options:
    disp = plot_confusion_matrix(knn, X_test, y_test,
                                 cmap=plt.cm.Blues,
                                 normalize=normalize)
    disp.ax_.set_title(title)

    print(title)
    print(disp.confusion_matrix)

plt.show()<jupyter_output>Confusion matrix, without normalization
[[115  36]
 [ 33  47]]
Normalized confusion matrix
[[0.7615894 0.2384106]
 [0.4125    0.5875   ]]<jupyter_text>** Print the classification report **<jupyter_code>print(classification_report(y_test, y_pred))<jupyter_output>precision    recall  f1-score   support

         0.0       0.78      0.76      0.77       151
         1.0       0.57      0.59      0.58        80

    accuracy                           0.70       231
   macro avg       0.67      0.67      0.67       231
weighted avg       0.70      0.70      0.70       231<jupyter_text>In the case of the K nearest neighbors algorithm, the K parameter is one of the most important parameters affecting the model performance.  The model performance isn't horrible, but what if we didn't consider a wide enough range of values in our neighbors for the KNN? An alternative to fitting a loop of models is to use a grid search to identify the proper number. It is common practice to use a grid search method for all adjustable parameters in any type of machine learning algorithm. First, you define the grid — aka the range of values — to test in the parameter being optimized, and then compare the model outcome performance based on the different values in the grid.  Run the code in the next cell to see how to implement the grid search method for identifying the best parameter value for the n_neighbors parameter. Notice the param_grid is the range value to test and we apply cross validation with five folds to score each possible value of n_neighbors.<jupyter_code>from sklearn.model_selection import GridSearchCV
param_grid = {'n_neighbors':np.arange(1,50)}
knn = KNeighborsClassifier()
knn_cv= GridSearchCV(knn,param_grid,cv=5)
knn_cv.fit(X_train,y_train)<jupyter_output><empty_output><jupyter_text>Print the best score and best parameter for n_neighbors.<jupyter_code>print("Best Score:" + str(knn_cv.best_score_))
print("Best Parameters: " + str(knn_cv.best_params_))<jupyter_output>Best Score:0.737417791623399
Best Parameters: {'n_neighbors': 24}<jupyter_text>Here you can see that the ideal number of n_neighbors for this model is 14 based on the grid search performed.  ** Now, following the KNN example, apply this grid search method to find the optimal number of estimators in a Randon Forest model.**<jupyter_code>from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import RandomizedSearchCV

param_grid = {'n_estimators':np.arange(50,200,50),
              'criterion':['gini', 'entropy'],
              'max_depth':np.arange(1,11,2),
              }
rfc = RandomForestClassifier()
rfc_cv= GridSearchCV(rfc,param_grid,cv=5)
rfc_cv.fit(X_train,y_train)
print("Best Score:" + str(rfc_cv.best_score_))
print("Best Parameters: " + str(rfc_cv.best_params_))<jupyter_output>Best Score:0.7783489096573208
Best Parameters: {'criterion': 'entropy', 'max_depth': 7, 'n_estimators': 100}
{'fraction_non_alphanumeric': 0.1004049423735853, 'fraction_numerical': 0.042778527671062196, 'mean_word_length': 4.545192861255037, 'pattern_counts': {'":': 0, '<': 84, '<?xml version=': 0, '>': 84, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '171233', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 3475, 'n_tokens_neox': 3125, 'n_words': 828, 'path': 'GridSearchKNN_Case_Study/GridSearchKNN_Case_Study.ipynb', 'repo_name': 'andrea-fantini/Springboard', 'size': '9631'}