<jupyter_start><jupyter_text>Experiment 1.2 - Xgboost with hyperparameter tuningUse a method to tune hyperparameters for the xgboost model.https://www.mikulskibartosz.name/xgboost-hyperparameter-tuning-in-python-using-grid-search/https://machinelearningmastery.com/tune-number-size-decision-trees-xgboost-python/<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
# import load_data function from helper file 
%load_ext autoreload
%autoreload 2

# fix system path
import sys
sys.path.append("/home/jovyan/work")
from src.features.helper_functions import load_sets

X_train, y_train, X_val, y_val, X_test = load_sets()
print(X_train)
print(X_val)<jupyter_output>[[ 7.416e+03  6.400e+01  1.390e+01 ...  4.000e-01  1.000e-01  7.000e-01]
 [ 4.919e+03  8.100e+01  2.080e+01 ...  6.000e-01  1.000e-01  1.300e+00]
 [ 7.672e+03  5.000e+01  5.600e+00 ...  4.000e-01 -3.000e-01  3.000e-01]
 ...
 [ 5.832e+03  5.500e+01  1.560e+01 ...  5.000e-01  3.000e-01  8.000e-01]
 [ 5.163e+03  9.100e+01  4.790e+01 ...  9.000e-01  6.000e-01  1.600e+00]
 [ 8.346e+03  6.700e+01  2.640e+01 ...  6.000e-01  2.400e+00  1.000e+00]]
[[9.0960e+03 4.2000e+01 7.4000e+00 ... 3.0000e-01 0.0000e+00 3.0000e-01]
 [1.1122e+04 5.9000e+01 1.2600e+01 ... 4.0000e-01 2.0000e-01 1.0000e+00]
 [1.1247e+04 7.0000e+01 1.6100e+01 ... 4.0000e-01 4.0000e-01 1.4000e+00]
 ...
 [1.0720e+04 6.7000e+01 2.3800e+01 ... 1.1000e+00 3.0000e-01 1.9000e+00]
 [7.6770e+03 4.4000e+01 2.0200e+01 ... 7.0000e-01 4.0000e-01 1.3000e+00]
 [1.1406e+04 7.9000e+01 2.8500e+01 ... 1.2000e+00 1.0000e-01 3.0000e+00]]<jupyter_text>Create grid search parameters<jupyter_code># pip install xgboost -- run this if xgboost doesn't load
from xgboost import XGBClassifier
from sklearn.model_selection import GridSearchCV
# instatiate xgboost 
model = XGBClassifier(seed=42)
# define parameters to try
parameters = {
    'max_depth': range (2, 10, 1),
    'n_estimators': range(60, 220, 40),
    'learning_rate': [0.1, 0.01, 0.05]
}
grid_search = GridSearchCV(
    estimator=model,
    param_grid=parameters,
    scoring = 'roc_auc', # use roc_auc to score
    n_jobs = -1, # use all available processors
    cv = 10, # no of folds for cross validation
    verbose=3 # display all info as the model trains
)
# train
grid_result = grid_search.fit(X_train, y_train)
# from sklearn.model_selection import StratifiedKFold
# kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=7)

# summarise results
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))<jupyter_output>Best: 0.699132 using {'learning_rate': 0.05, 'max_depth': 2, 'n_estimators': 140}<jupyter_text>Train the model using best parameters<jupyter_code># Train using best parameters
model = XGBClassifier(max_depth = 2, n_estimators = 140, learning_rate = 0.05)
model_best_params = model.fit(X_train, y_train)
from src.features.helper_functions import save_model
save_model(model_best_params, 'rex_xgb_gridsearch')<jupyter_output>Model saved<jupyter_text>Make predictions using validation set<jupyter_code># predict class
y_train_preds = model_best_params.predict(X_train)
y_val_preds = model_best_params.predict(X_val)
# predict proabilities
y_train_preds_prob = model_best_params.predict_proba(X_train)
y_val_preds_prob = model_best_params.predict_proba(X_val)
from sklearn.metrics import mean_squared_error as mse
from sklearn.metrics import mean_absolute_error as mae
print(mse(y_train, y_train_preds, squared=False))
print(mae(y_train, y_train_preds))

print(mse(y_val, y_val_preds, squared=False))
print(mae(y_val, y_val_preds))
from sklearn.metrics import accuracy_score

accuracy = accuracy_score(y_val, y_val_preds)
print("Accuracy: %.2f%%" % (accuracy * 100.0))
from sklearn.metrics import roc_auc_score ,recall_score, precision_score
precision=precision_score(y_val, y_val_preds)
recall=recall_score(y_val, y_val_preds)
roc=roc_auc_score(y_val, y_val_preds)
print("Precision: %.2f%% " % (precision *100))
print("Recall: %.2f%% " % (recall * 100))
print("AUC: %.2f%% " % (roc *100))
roc_auc_score(y_train, y_train_preds_prob[:,1])
roc_auc_score(y_val, y_val_preds_prob[:,1])
from sklearn.metrics import plot_confusion_matrix

plot_confusion_matrix(model_best_params, X_val, y_val, cmap=plt.cm.Blues)  
plt.show() 
from sklearn.metrics import classification_report

class_report = classification_report(y_val, y_val_preds)
print(class_report)<jupyter_output>precision    recall  f1-score   support

           0       0.31      0.02      0.04       257
           1       0.84      0.99      0.91      1343

    accuracy                           0.84      1600
   macro avg       0.58      0.51      0.47      1600
weighted avg       0.76      0.84      0.77      1600<jupyter_text>About 250 are being falsely classed as 1, whereas only 11 are being wrongly classed as 0. This may be due to the imbalance majority classs 1, overpowering the 0's.  Plot roc curve using code from here https://www.analyticsvidhya.com/blog/2020/06/auc-roc-curve-machine-learning/<jupyter_code>from sklearn.metrics import roc_curve

# roc curve for models
fpr, tpr, thresh = roc_curve(y_val, y_val_preds_prob[:,1], pos_label=1)

# roc curve for tpr = fpr 
random_probs = [0 for i in range(len(y_val))]
p_fpr, p_tpr, _ = roc_curve(y_val, random_probs, pos_label=1)
# use matplotlib to plot ROC curve
plt.style.use('seaborn')

# plot roc curves
plt.plot(fpr, tpr, linestyle='--',color='orange', label='XGBoost best params')
plt.plot(p_fpr, p_tpr, linestyle='--', color='blue')
# title
plt.title('ROC curve')
# x label
plt.xlabel('False Positive Rate')
# y label
plt.ylabel('True Positive rate')

plt.legend(loc='best')
plt.savefig('../src/visualisation/Rez XGBoost best params',dpi=300)
plt.show()<jupyter_output><empty_output><jupyter_text>Slight improvement over base model, the AUC is higher at 0.702, but that is still under the current best score on the leaderboard.More paramaters can be tuned, try a few more before moving onto a different model e.g. SVM.  Make predictions with test set for Kaggle submission<jupyter_code>y_test_preds = model_best_params.predict(X_test)
y_test_preds_prob = model_best_params.predict_proba(X_test)
print(y_test_preds)
print(y_test_preds_prob)
unique_elements, counts_elements = np.unique(y_test_preds, return_counts=True)
print(np.asarray((unique_elements, counts_elements)))
# convert X_test back to df
df_test = pd.DataFrame(X_test)
print(df_test)
# output predictions
id_col = df_test.iloc[:,[0]]
print(id_col)
id_col.rename(columns = {0:'Id'}, inplace = True)
print(id_col.columns)
print(id_col.info())
# need to change Id to int
id_col = id_col.Id.astype(int)
probabilities = pd.DataFrame(y_test_preds_prob[:,1], columns = ['TARGET_5Yrs'])
print(probabilities)
# concat columns
output = pd.concat([id_col,probabilities], axis=1)

#df_output = pd.DataFrame(output_list, columns = ['Id','TARGET_5Yrs'])
print(output.head())
# save to csv
output.to_csv('../data/processed/output_xgboost_best_params_wk1.csv',index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1241114586295138, 'fraction_numerical': 0.08999147000284333, 'mean_word_length': 4.785361842105263, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '494135', 'license': 'FTL', 'n_tokens_mistral': 3150, 'n_tokens_neox': 2732, 'n_words': 584, 'path': 'notebooks/tith_reasmey-10845345-week1_xgboosthptuning.ipynb', 'repo_name': 'cvbilt/adsi_group2leftfeet', 'size': '7034'}