<jupyter_start><jupyter_text>Lab 3: Model Optimisation Exercise 2: Xgboost with HyperoptWe will train a Xgboost model on the same dataset as previously usiong Hyperopt.**Pre-requisites:**- Create a github account (https://github.com/join)- Install git (https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)- Install Docker (https://docs.docker.com/get-docker/)The steps are:1.   Launch Docker image2.   Load Data3.   Train Xgboost model with defauly hyperparameter4.   Hyperparameter tuning with Hyperopt5.   Push changes<jupyter_code>"""

# Launch Docker image
docker run  -dit --rm --name adv_dsi_lab_3 -p 8888:8888 -e JUPYTER_ENABLE_LAB=yes -v ~/Projects/adv_dsi/adv_dsi_lab_3:/home/jovyan/work -v ~/.aws:/home/jovyan/.aws -v ~/Projects/adv_dsi/src:/home/jovyan/work/src xgboost-notebook:latest 
docker logs --tail 50 adv_dsi_lab_2

# Create a new git branch called xgboost_hyperopt
git checkout -b xgboost_hyperopt

# Navigate the folder notebooks and create a new jupyter notebook called 2_xgboost_hyperopt.ipynb

"""<jupyter_output><empty_output><jupyter_text>2. Load Data**[2.1]** Import the function you created `load_sets` from `src/data/sets`**[2.2]** Load the saved sets from `data/processed`<jupyter_code># Import the function you created load_sets from src/data/sets
from src.data.sets import load_sets

# Load the saved sets from data/processed
X_train, y_train, X_val, y_val, X_test, y_test = load_sets(path='../data/processed/')<jupyter_output><empty_output><jupyter_text>3. Train Xgboost model**[3.1]** Import the xgboost package as xgb**[3.2]** Instantiate the RandomForest class into a variable called rf with random_state=8**[3.3]** Task: Fit the model with the prepared data**[3.4]** Import `dump` from `joblib` and save the fitted model into the folder `models` as a file called `xgboost_default`**[3.5]** Save the predictions from this model for the training and validation sets into 2 variables called `y_train_preds` and `y_val_preds`**[3.6]** Import `print_reg_perf` from `src/models/performance` and display the accuracy and f1 scores of this baseline model on the training and validation sets<jupyter_code># Import the xgboost package as xgb
import xgboost as xgb

# instantiate the XGBClassifier class into a variable called xgboost1
xgboost1 = xgb.XGBClassifier()

# Fit the model with the prepared data
xgboost1.fit(X_train, y_train)
print(xgboost1.fit(X_train, y_train))

# Import dump from joblib and save the fitted model into the folder models as a file called xgboost_default
from joblib import dump 
dump(xgboost1,  '../models/xgboost_default.joblib')

# Save the predictions from this model for the training and validation sets into 2 variables called y_train_preds and y_val_preds
y_train_preds = xgboost1.predict(X_train)
y_val_preds = xgboost1.predict(X_val)

from src.models.performance import print_class_perf

print_class_perf(y_preds=y_train_preds, y_actuals=y_train, set_name='Training', average='weighted')
print_class_perf(y_preds=y_val_preds, y_actuals=y_val, set_name='Validation', average='weighted')<jupyter_output>Accuracy Training: 0.9211552214945801
F1 Training      : 0.9208732214873642
Accuracy Validation: 0.9068593424622957
F1 Validation      : 0.9063573849035116<jupyter_text>4. Hyperparameter tuning with Hyperopt**[4.1]** Import Trials, STATUS_OK, tpe, hp, fmin from hyperopt package**[4.2]** Define the search space for xgboost hyperparameters<jupyter_code># Import Trials, STATUS_OK, tpe, hp, fmin from hyperopt package
from hyperopt import Trials, STATUS_OK, tpe, hp, fmin

# Define the search space for xgboost hyperparameters
space = {
    'max_depth' : hp.choice('max_depth', range(5, 20, 1)),
    'learning_rate' : hp.quniform('learning_rate', 0.01, 0.5, 0.05),
    'min_child_weight' : hp.quniform('min_child_weight', 1, 10, 1),
    'subsample' : hp.quniform('subsample', 0.1, 1, 0.05),
    'colsample_bytree' : hp.quniform('colsample_bytree', 0.1, 1.0, 0.05)
}<jupyter_output><empty_output><jupyter_text>**[4.3]** Define a function called `objective` with the following logics:- input parameters: hyperparameter seacrh space (`space`)- logics: train a xgboost model with the search space and calculate the average accuracy score for cross validation with 10 folds- output parameters: dictionary with the loss score and STATUS_OK<jupyter_code># Define a function called objective
def objective(space):
    from sklearn.model_selection import cross_val_score
    
    xgboost = xgb.XGBClassifier(
        max_depth = int(space['max_depth']),
        learning_rate = space['learning_rate'],
        min_child_weight = space['min_child_weight'],
        subsample = space['subsample'],
        colsample_bytree = space['colsample_bytree']
    )
    
    acc = cross_val_score(xgboost, X_train, y_train, cv=10, scoring="accuracy").mean()

    return{'loss': 1-acc, 'status': STATUS_OK }<jupyter_output><empty_output><jupyter_text>**[4.4]** Launch Hyperopt search and save the result in a variable called `best`<jupyter_code># Launch Hyperopt search and save the result in a variable called `best`
best = fmin(
    fn=objective,   
    space=space,       
    algo=tpe.suggest,       
    max_evals=5
)<jupyter_output>100%|██████████| 5/5 [23:15<00:00, 279.19s/trial, best loss: 0.1020389329003667]<jupyter_text>**[4.5]** Print the best set of hyperparameters<jupyter_code># Print the best set of hyperparameters
print("Best: ", best)<jupyter_output>Best:  {'colsample_bytree': 0.9500000000000001, 'learning_rate': 0.25, 'max_depth': 8, 'min_child_weight': 4.0, 'subsample': 0.25}<jupyter_text>**[4.6]** Instantiate a XGBClassifier with best set of hyperparameters**[4.7]** Fit the model with the prepared data<jupyter_code># Instantiate a XGBClassifier with best set of hyperparameters
xgboost2 = xgb.XGBClassifier(
    max_depth = best['max_depth'],
    learning_rate = best['learning_rate'],
    min_child_weight = best['min_child_weight'],
    subsample = best['subsample'],
    colsample_bytree = best['colsample_bytree']
)

# Fit the model with the prepared data
xgboost2.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>**[4.8]** Display the accuracy and f1 scores of this baseline model on the training and validation sets<jupyter_code># Display the accuracy and f1 scores of this baseline model on the training and validation sets
print_class_perf(y_preds=xgboost2.predict(X_train), y_actuals=y_train, set_name='Training', average='weighted')
print_class_perf(y_preds=xgboost2.predict(X_val), y_actuals=y_val, set_name='Validation', average='weighted')<jupyter_output>Accuracy Training: 0.9243174264839926
F1 Training      : 0.9240612030551106
Accuracy Validation: 0.9039776193180623
F1 Validation      : 0.9035708202543029<jupyter_text>**[4.9]** Save the fitted model into the folder models as a file called `xgboost_best`<jupyter_code># Save the fitted model into the folder models as a file called `xgboost_best`
dump(xgboost2,  '../models/xgboost_best.joblib')<jupyter_output><empty_output><jupyter_text>5. Push changes<jupyter_code>"""
# Add you changes to git staging area
# Create the snapshot of your repository and add a description
# Push your snapshot to Github

git add .   
git commit -m "commit 3 xgboost"
git push https://*********@github.com/CazMayhem/adv_dsi_lab_3.git

# Check out to the master branch
# Pull the latest updates

git checkout master
git pull https://*********@github.com/CazMayhem/adv_dsi_lab_3.git

# Check out to the data_prep branch
# Merge the master branch and push your changes
git checkout xgboost_hyperopt
git merge master
git push https://*********@github.com/CazMayhem/adv_dsi_lab_3.git

"""<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10842901878914404, 'fraction_numerical': 0.04149269311064718, 'mean_word_length': 5.551282051282051, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 41, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '170292', 'license': 'FTL', 'n_tokens_mistral': 2822, 'n_tokens_neox': 2586, 'n_words': 739, 'path': 'notebooks/2_xgboost_hyperopt.ipynb', 'repo_name': 'CazMayhem/adv_dsi_lab_3', 'size': '7664'}