<jupyter_start><jupyter_text>Grid Search Hyperparameter optimization This case study is all about using grid searches to identify the optimal parameters for a machine learning algorithm. To complere this case study, you'll use the Pima Indian diabetes dataset from Kaggle and KNN. Follow along with the preprocessing steps of this case study. Load the necessary packages<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import confusion_matrix
from sklearn.metrics import plot_confusion_matrix
from sklearn.metrics import classification_report
from sklearn.ensemble import RandomForestClassifier

sns.set()

# set random seed to try make this exercise and solutions reproducible (NB: this is just for teaching purpose and not something you would do in real life)
random_seed_number = 42
np.random.seed(random_seed_number)<jupyter_output>Bad key "text.kerning_factor" on line 4 in
C:\Users\gosal\Anaconda3_2\lib\site-packages\matplotlib\mpl-data\stylelib\_classic_test_patch.mplstyle.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution<jupyter_text>Load the diabetes data<jupyter_code>diabetes_data = pd.read_csv('data/diabetes.csv')
diabetes_data.head()<jupyter_output><empty_output><jupyter_text>** Start by reviewing the data info.**<jupyter_code>diabetes_data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 768 entries, 0 to 767
Data columns (total 9 columns):
 #   Column                    Non-Null Count  Dtype  
---  ------                    --------------  -----  
 0   Pregnancies               768 non-null    int64  
 1   Glucose                   768 non-null    int64  
 2   BloodPressure             768 non-null    int64  
 3   SkinThickness             768 non-null    int64  
 4   Insulin                   768 non-null    int64  
 5   BMI                       768 non-null    float64
 6   DiabetesPedigreeFunction  768 non-null    float64
 7   Age                       768 non-null    int64  
 8   Outcome                   768 non-null    int64  
dtypes: float64(2), int64(7)
memory usage: 54.1 KB<jupyter_text>** Apply the describe function to the data.**<jupyter_code>diabetes_data.describe()
diabetes_data.isnull().sum()<jupyter_output><empty_output><jupyter_text>** Currently, the missing values in the dataset are represented as zeros. Replace the zero values in the following columns ['Glucose','BloodPressure','SkinThickness','Insulin','BMI'] with nan .**<jupyter_code>diabetes_data[['Glucose','BloodPressure','SkinThickness','Insulin','BMI']] = diabetes_data[['Glucose','BloodPressure','SkinThickness','Insulin','BMI']].replace(0,np.nan)
diabetes_data.isnull().sum()<jupyter_output><empty_output><jupyter_text>** Plot histograms of each column. **<jupyter_code>for i in ['Glucose','BloodPressure','SkinThickness','Insulin','BMI']:
    _ = diabetes_data.hist(i)
    plt.show()<jupyter_output><empty_output><jupyter_text>Replace the zeros with mean and median values.<jupyter_code>diabetes_data['Glucose'].fillna(diabetes_data['Glucose'].mean(), inplace = True)
diabetes_data['BloodPressure'].fillna(diabetes_data['BloodPressure'].mean(), inplace = True)
diabetes_data['SkinThickness'].fillna(diabetes_data['SkinThickness'].median(), inplace = True)
diabetes_data['Insulin'].fillna(diabetes_data['Insulin'].median(), inplace = True)
diabetes_data['BMI'].fillna(diabetes_data['BMI'].median(), inplace = True)
diabetes_data.isnull().sum()<jupyter_output><empty_output><jupyter_text>** Plot histograms of each column after replacing nan. **<jupyter_code>for i in ['Glucose','BloodPressure','SkinThickness','Insulin','BMI']:
    _ = diabetes_data.hist(i)
    plt.show()<jupyter_output><empty_output><jupyter_text>Plot the correlation matrix heatmap<jupyter_code>plt.figure(figsize=(12,10))
print('Correlation between various features')
p=sns.heatmap(diabetes_data.corr(), annot=True,cmap ='Blues')<jupyter_output>Correlation between various features<jupyter_text>** Define the `y` variable as the `Outcome` column.**<jupyter_code>X = diabetes_data.drop('Outcome', axis = 1)

y = diabetes_data['Outcome']<jupyter_output><empty_output><jupyter_text>** Create a 70/30 train and test split. ** ** Using Sklearn, standarize the magnitude of the features by scaling the values. ** Note: Don't forget to fit() your scaler on X_train and then use that fitted scaler to transform() X_test. This is to avoid data leakage while you standardize your data.<jupyter_code>scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.3, random_state=42)<jupyter_output><empty_output><jupyter_text>Using a range of neighbor values of 1-10, apply the KNearestNeighbor classifier to classify the the data.<jupyter_code>test_scores = []
train_scores = []

for i in range(1,10):

    knn = KNeighborsClassifier(i)
    knn.fit(X_train,y_train)
    
    train_scores.append(knn.score(X_train,y_train))
    test_scores.append(knn.score(X_test,y_test))<jupyter_output><empty_output><jupyter_text>** Print the train and test scores for each iteration.**<jupyter_code>for i in range(1,10):
    print('Train score is {}'.format(train_scores[i-1]), 'Test score is {}'.format(test_scores[i-1]))<jupyter_output>Train score is 1.0 Test score is 0.6666666666666666
Train score is 0.8435754189944135 Test score is 0.670995670995671
Train score is 0.8547486033519553 Test score is 0.6666666666666666
Train score is 0.8379888268156425 Test score is 0.683982683982684
Train score is 0.8379888268156425 Test score is 0.7186147186147186
Train score is 0.819366852886406 Test score is 0.6883116883116883
Train score is 0.8324022346368715 Test score is 0.7056277056277056
Train score is 0.819366852886406 Test score is 0.7402597402597403
Train score is 0.819366852886406 Test score is 0.7272727272727273<jupyter_text>** Identify the number of neighbors that resulted in the max score in the training and testing dataset. **<jupyter_code>knn = KNeighborsClassifier(3)
knn.fit(X_train, y_train)
print("n_neighbors = 3")
print(f"Train score: {knn.score(X_train, y_train)}")
print(f"Test score: {knn.score(X_test, y_test)}")
def train_KNN(n_neighbors):
    knn = KNeighborsClassifier(n_neighbors = n)
    knn_model = knn.fit(X_train, y_train)
    train_score = knn.score(X_train, y_train)
    test_score = knn.score(X_test, y_test)
    print('Train score: {} / Test score: {} for {} neighbors'.format(train_score, test_score, n))
for n in range(1,10):
    train_KNN(n)<jupyter_output>Train score: 1.0 / Test score: 0.6666666666666666 for 1 neighbors
Train score: 0.8435754189944135 / Test score: 0.670995670995671 for 2 neighbors
Train score: 0.8547486033519553 / Test score: 0.6666666666666666 for 3 neighbors
Train score: 0.8379888268156425 / Test score: 0.683982683982684 for 4 neighbors
Train score: 0.8379888268156425 / Test score: 0.7186147186147186 for 5 neighbors
Train score: 0.819366852886406 / Test score: 0.6883116883116883 for 6 neighbors
Train score: 0.8324022346368715 / Test score: 0.7056277056277056 for 7 neighbors
Train score: 0.819366852886406 / Test score: 0.7402597402597403 for 8 neighbors
Train score: 0.819366852886406 / Test score: 0.7272727272727273 for 9 neighbors<jupyter_text>Max score in training dataset was 0.854 for 3 neighborsMax score in testing dataset was 0.740 for 8 neighbors Plot the train and test model performance by number of neighbors.<jupyter_code>plt.figure(figsize=(12,5))
p = sns.lineplot(range(1,10),train_scores,marker='*',label='Train Score')
p = sns.lineplot(range(1,10),test_scores,marker='o',label='Test Score')<jupyter_output><empty_output><jupyter_text>** Fit and score the best number of neighbors based on the plot. **<jupyter_code>knn = KNeighborsClassifier(n_neighbors = 8)
knn_model = knn.fit(X_train, y_train)
train_score = knn.score(X_train, y_train)
test_score = knn.score(X_test, y_test)
print('Train score: {} / Test score: {} for 8 neighbors'.format(train_score, test_score))
y_pred = knn.predict(X_test)
pl = confusion_matrix(y_test,y_pred)
pl<jupyter_output><empty_output><jupyter_text>** Plot the confusion matrix for the model fit above. **<jupyter_code>_ = plot_confusion_matrix(knn, X_test, y_test) 
_ = plt.title('Diabetes - 1, No Diabetes - 0')
_ = plt.show()<jupyter_output><empty_output><jupyter_text>** Print the classification report **<jupyter_code>print(classification_report(y_test, y_pred))<jupyter_output>precision    recall  f1-score   support

           0       0.78      0.85      0.81       151
           1       0.65      0.54      0.59        80

    accuracy                           0.74       231
   macro avg       0.71      0.69      0.70       231
weighted avg       0.73      0.74      0.73       231<jupyter_text>In the case of the K nearest neighbors algorithm, the K parameter is one of the most important parameters affecting the model performance.  The model performance isn't horrible, but what if we didn't consider a wide enough range of values in our neighbors for the KNN? An alternative to fitting a loop of models is to use a grid search to identify the proper number. It is common practice to use a grid search method for all adjustable parameters in any type of machine learning algorithm. First, you define the grid — aka the range of values — to test in the parameter being optimized, and then compare the model outcome performance based on the different values in the grid.  Run the code in the next cell to see how to implement the grid search method for identifying the best parameter value for the n_neighbors parameter. Notice the param_grid is the range value to test and we apply cross validation with five folds to score each possible value of n_neighbors.<jupyter_code>from sklearn.model_selection import GridSearchCV
param_grid = {'n_neighbors':np.arange(1,50)}
knn = KNeighborsClassifier()
knn_cv= GridSearchCV(knn,param_grid,cv=5)
knn_cv.fit(X,y)<jupyter_output><empty_output><jupyter_text>Print the best score and best parameter for n_neighbors.<jupyter_code>print("Best Score:" + str(knn_cv.best_score_))
print("Best Parameters: " + str(knn_cv.best_params_))<jupyter_output>Best Score:0.7526440879382056
Best Parameters: {'n_neighbors': 31}<jupyter_text>Here you can see that the ideal number of n_neighbors for this model is 14 based on the grid search performed.  ** Now, following the KNN example, apply this grid search method to find the optimal number of estimators in a Randon Forest model.**<jupyter_code>param_grid = { 
    'n_estimators': np.arange(1,50),
    'max_features': ['auto', 'sqrt', 'log2'],
    'max_depth' : [4,5,6,7,8],
    'criterion' :['gini', 'entropy']
}

rfc = RandomForestClassifier()
CV_rfc = GridSearchCV(rfc, param_grid, cv= 5)
CV_rfc.fit(X,y)
print("Best Score: " + str(CV_rfc.best_score_))
print("Best Parameters: " + str(CV_rfc.best_params_))<jupyter_output>Best Score: 0.7852134793311263
Best Parameters: {'criterion': 'entropy', 'max_depth': 7, 'max_features': 'sqrt', 'n_estimators': 41}
{'fraction_non_alphanumeric': 0.09942452412571934, 'fraction_numerical': 0.07268702965914121, 'mean_word_length': 4.9862215156332805, 'pattern_counts': {'":': 0, '<': 78, '<?xml version=': 0, '>': 78, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 4, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '509442', 'license': 'MIT', 'n_tokens_mistral': 4323, 'n_tokens_neox': 3671, 'n_words': 1052, 'path': 'Case study - Grid Search in KNN/GridSearchKNN_Case_Study.ipynb', 'repo_name': 'shalin4788/Springboard-Do-not-refer-', 'size': '11295'}