<jupyter_start><jupyter_text>Hyper-parameter optimisation HPO is the process of finding a set of optimal hyperparameters for a learning system . HPO is an essential step in ensuring that the final model generalizes well in different situations . We are going to optimize our hyperparameters by using the following methods:


1.   Bayesian optimisation
2.   HPO using TPE (Hyperopt)
3.   HPO using Optuna

---

 Importing essentials<jupyter_code>import pandas as pd 
import numpy as np
import pickle as pk
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.ensemble import StackingClassifier
from sklearn.svm import SVC
from sklearn import model_selection
from sklearn import metrics
from skopt import space,gp_minimize
from functools import partial
from hyperopt import hp,fmin,tpe,Trials
from hyperopt.pyll.base import scope
import optuna<jupyter_output><empty_output><jupyter_text>Loading the training data and the test data<jupyter_code>with open("X_re.pickle","rb") as f:
  X_train=pk.load(f)
with open('y_re.pickle',"rb") as f:
  y_train=pk.load(f)
with open("X_test.pickle","rb") as f:
  X_test=pk.load(f)
with open('y_test.pickle',"rb") as f:
  y_test=pk.load(f)  
def get_models():
  models=list()
  models.append(("lr",LogisticRegression()))
  models.append(("rf",RandomForestClassifier()))
  models.append(("nb",GaussianNB()))
  models.append(("svm",SVC()))
  return models<jupyter_output><empty_output><jupyter_text>Bayesian Optimisation  Optimize function<jupyter_code>def optimize(params,param_names,x,y,base_models):
  params=dict(zip(param_names,params))
  stacked_model=RandomForestClassifier(**params)
  model=StackingClassifier(estimators=base_models,final_estimator=stacked_model,cv=5)
  kf=model_selection.StratifiedKFold(n_splits=5)
  accuracies=[] #accuracies for all the fold
  for idx in kf.split(X=x,y=y):
    train_idx,test_idx=idx[0],idx[1]

    #train split
    xtrain=x[train_idx]
    ytrain=y[train_idx]

    #test split
    xtest=x[test_idx]
    ytest=y[test_idx]

    model.fit(xtrain,ytrain) #fitting the stacked model
    preds=model.predict_proba(xtest)
    preds=preds[:,1]
    fold_acc=metrics.roc_auc_score(ytest,preds)
    accuracies.append(fold_acc) 
  
  return -1.0*np.mean(accuracies)<jupyter_output><empty_output><jupyter_text>Defining the parameter space and the parameter names<jupyter_code>param_space=[
             space.Integer(3,10,name="max_depth"),
             space.Categorical(["gini","entropy"],name="criterion"),
             space.Integer(100,500,name="n_estimators"),
             space.Real(0.01,1,prior="uniform",name="max_features")
]
param_names=[
             "max_depth",
             "criterion",
             "n_estimators",
             "max_features"
]<jupyter_output><empty_output><jupyter_text>Now we do optimization using gp_minimize<jupyter_code>optmization_function=partial(optimize,
                             param_names=param_names,
                             x=X_train,
                             y=y_train,
                             base_models=get_models())
result=gp_minimize(optmization_function,
                   dimensions=param_space,
                   n_calls=5,
                   n_random_starts=5,
                   verbose=10,
                   n_jobs=-1)<jupyter_output>Iteration No: 1 started. Evaluating function at random point.
Iteration No: 1 ended. Evaluation done at random point.
Time taken: 40.7732
Function value obtained: -0.9894
Current minimum: -0.9894
Iteration No: 2 started. Evaluating function at random point.
Iteration No: 2 ended. Evaluation done at random point.
Time taken: 36.2635
Function value obtained: -0.9888
Current minimum: -0.9894
Iteration No: 3 started. Evaluating function at random point.
Iteration No: 3 ended. Evaluation done at random point.
Time taken: 43.5193
Function value obtained: -0.9900
Current minimum: -0.9900
Iteration No: 4 started. Evaluating function at random point.
Iteration No: 4 ended. Evaluation done at random point.
Time taken: 43.8138
Function value obtained: -0.9901
Current minimum: -0.9901
Iteration No: 5 started. Evaluating function at random point.
Iteration No: 5 ended. Evaluation done at random point.
Time taken: 49.6882
Function value obtained: -0.9885
Current minimum: -0.9901<jupyter_text>Final results<jupyter_code>print(dict(zip(param_names,result.x)))<jupyter_output>{'max_depth': 3, 'criterion': 'entropy', 'n_estimators': 295, 'max_features': 0.7979762924139232}<jupyter_text>HPO using Hyperopt Defining the optimize function<jupyter_code>def optimize(params,x,y,base_models):
  stacked_model=RandomForestClassifier(**params)
  model=StackingClassifier(estimators=base_models,final_estimator=stacked_model,cv=5)
  kf=model_selection.StratifiedKFold(n_splits=5)
  accuracies=[] #accuracies for all the fold
  for idx in kf.split(X=x,y=y):
    train_idx,test_idx=idx[0],idx[1]

    #train split
    xtrain=x[train_idx]
    ytrain=y[train_idx]

    #test split
    xtest=x[test_idx]
    ytest=y[test_idx]

    model.fit(xtrain,ytrain) #fitting the stacked model
    preds=model.predict_proba(xtest)
    preds=preds[:,1]
    fold_acc=metrics.roc_auc_score(ytest,preds)
    accuracies.append(fold_acc) 
  
  return -1.0*np.mean(accuracies)<jupyter_output><empty_output><jupyter_text>Creating the parameter space<jupyter_code>param_space={
    "max_depth":scope.int(hp.quniform("max_depth",3,10,1)),
    "criterion":hp.choice("criterion",["gini","entropy"]),
    "n_estimators":scope.int(hp.quniform("n_estimators",100,500,1)),
    "max_features":hp.uniform("max_features",0.01,1)
}<jupyter_output><empty_output><jupyter_text>Now we optimize using fmin from Hyperopt<jupyter_code>optmization_function=partial(optimize,
                             x=X_train,
                             y=y_train,
                             base_models=get_models())
trials=Trials()
result=fmin(fn=optmization_function,
            space=param_space,
            algo=tpe.suggest,
            max_evals=15,
            trials=trials,
            
)<jupyter_output>100%|██████████| 15/15 [10:30<00:00, 42.04s/it, best loss: -0.9898109728841039]<jupyter_text>Final results<jupyter_code>print(result)<jupyter_output>{'criterion': 0, 'max_depth': 10.0, 'max_features': 0.24223551097941823, 'n_estimators': 141.0}<jupyter_text>HPO using Optuna Defining the optimize function<jupyter_code>def optimize(trial,x,y,base_models):
  criterion=trial.suggest_categorical("criterion",["gini","entropy"])
  n_estimators=trial.suggest_int("n_estimators",100,500)
  max_depth=trial.suggest_int("max_depth",3,10)
  max_features=trial.suggest_uniform("max_features",0.01,1.0)

  stacked_model=RandomForestClassifier(
      n_estimators=n_estimators,
      max_depth=max_depth,
      max_features=max_features,
      criterion=criterion
  )
  model=StackingClassifier(estimators=base_models,final_estimator=stacked_model,cv=5)
  kf=model_selection.StratifiedKFold(n_splits=5)
  accuracies=[] #accuracies for all the fold
  for idx in kf.split(X=x,y=y):
    train_idx,test_idx=idx[0],idx[1]

    #train split
    xtrain=x[train_idx]
    ytrain=y[train_idx]

    #test split
    xtest=x[test_idx]
    ytest=y[test_idx]

    model.fit(xtrain,ytrain) #fitting the stacked model
    preds=model.predict_proba(xtest)
    preds=preds[:,1]
    fold_acc=metrics.roc_auc_score(ytest,preds)
    accuracies.append(fold_acc) 
  
  return -1.0*np.mean(accuracies)<jupyter_output><empty_output><jupyter_text>Now we optimize using Optuna<jupyter_code>optimization_function=partial(optimize,x=X_train,y=y_train,base_models=get_models())
study= optuna.create_study(direction="minimize")<jupyter_output>[32m[I 2021-02-17 11:48:25,987][0m A new study created in memory with name: no-name-5f1fe833-7ad5-4165-85e4-c728fe7b0265[0m<jupyter_text>Final results<jupyter_code>study.optimize(optimization_function,n_trials=15)<jupyter_output>[32m[I 2021-02-17 11:49:09,951][0m Trial 0 finished with value: -0.9892223640690595 and parameters: {'criterion': 'entropy', 'n_estimators': 477, 'max_depth': 4, 'max_features': 0.33666211879018004}. Best is trial 0 with value: -0.9892223640690595.[0m
[32m[I 2021-02-17 11:49:59,559][0m Trial 1 finished with value: -0.987952647990026 and parameters: {'criterion': 'gini', 'n_estimators': 360, 'max_depth': 10, 'max_features': 0.9883961287774781}. Best is trial 0 with value: -0.9892223640690595.[0m
[32m[I 2021-02-17 11:50:39,965][0m Trial 2 finished with value: -0.9894500909587582 and parameters: {'criterion': 'gini', 'n_estimators': 337, 'max_depth': 6, 'max_features': 0.04242281336983184}. Best is trial 2 with value: -0.9894500909587582.[0m
[32m[I 2021-02-17 11:51:21,928][0m Trial 3 finished with value: -0.9902304278723563 and parameters: {'criterion': 'entropy', 'n_estimators': 324, 'max_depth': 6, 'max_features': 0.47622987153704094}. Best is trial 3 with value: -0.990230427[...]<jupyter_text>Evaluating the optimized model Finally , it's time to evaluate our final model and see it performance .<jupyter_code>import matplotlib.pyplot as plt
from sklearn.metrics import roc_curve
from sklearn.metrics import roc_auc_score<jupyter_output><empty_output><jupyter_text>Now that we have completed HPO , we are going to use the hyperparameters suggested by **Optuna**<jupyter_code>meta_model=RandomForestClassifier(criterion= 'entropy', n_estimators= 324, max_depth= 6, max_features=0.476223)
stacked=StackingClassifier(estimators=get_models(),final_estimator=meta_model,cv=10)
stacked.fit(X_train,y_train)<jupyter_output><empty_output><jupyter_text>Using confusion matrix and k-fold cross validation Making a confusion matrix<jupyter_code>from sklearn.metrics import confusion_matrix
confusion_matrix(y_test,stacked.predict(X_test))<jupyter_output><empty_output><jupyter_text>K-fold cross validation<jupyter_code>from sklearn.model_selection import cross_val_score
scores=cross_val_score(stacked,X_train,y_train,cv=10)
scores
print("%0.2f accuracy with a standard deviation of %0.2f" % (scores.mean(), scores.std()))<jupyter_output>0.97 accuracy with a standard deviation of 0.05<jupyter_text>Using AUC and ROC Defining the function to plot the ROC curve<jupyter_code>def plot_roc_curve(fpr, tpr):
  # tpr=True Positive Rate
  # fpr=False Positive Rate
    plt.plot(fpr, tpr, color='orange', label='ROC')
    plt.plot([0, 1], [0, 1], color='darkblue', linestyle='--') #diagonal line
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.title('Receiver Operating Characteristic (ROC) Curve')
    plt.legend()
    plt.show()<jupyter_output><empty_output><jupyter_text>Predicting the probablities of the testing set<jupyter_code>probs=stacked.predict_proba(X_test)
prob_positive=probs[:,1] # keeping the probablities of the positive class only<jupyter_output><empty_output><jupyter_text>**Area under the curve (AUC)**<jupyter_code>auc=roc_auc_score(y_test,prob_positive)
print("AUC:%.2f"% auc)<jupyter_output>AUC:0.83<jupyter_text>**ROC curve**<jupyter_code>fpr, tpr, thresholds = roc_curve(y_test, prob_positive)
plot_roc_curve(fpr, tpr)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11464124966904951, 'fraction_numerical': 0.05577618921542671, 'mean_word_length': 4.895941727367326, 'pattern_counts': {'":': 4, '<': 78, '<?xml version=': 0, '>': 77, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '260577', 'license': 'MIT', 'n_tokens_mistral': 4488, 'n_tokens_neox': 3979, 'n_words': 775, 'path': 'Ensemble Learning/Stroke Prediction/StrokePrediction_HPO_and_Evaluation.ipynb', 'repo_name': 'mehulfollytobevice/MachineLearning', 'size': '11331'}