<jupyter_start><jupyter_text>Xgboost Classifier ```{tip}It is recommended to use google colaboratory to run this notebook``` Xgboost stands for "Extreme Gradient Boosting". This supervised algorithm is created using the principle of supervised learning, decision trees ensembles and tree boosting. The library was designed to provide good scalability, portability and accuracy.<jupyter_code># Extra libraries required

# Install ray tune
! pip install tune-sklearn ray[tune]

# Install shap
! pip install shap
# Import necessary packages
import pandas as pd
from imblearn.over_sampling import SMOTE
from sklearn.model_selection import StratifiedKFold, train_test_split
from xgboost import XGBClassifier, Booster
from sklearn.metrics import classification_report
import plotly.express as px
import plotly.io as pio
# Set default plotly renderer
pio.renderers.default = "notebook_connected" # Use "colab" when running in google colaboratory
# Load data into dataframe
df = pd.read_csv('/content/drive/MyDrive/Colab Notebooks/uci/ospi/datasets/preprocessed_osi.csv')<jupyter_output><empty_output><jupyter_text>Preproessing The preprocessing are the same as used for other ensembling methods.<jupyter_code>y = df['Revenue']
X = df.drop('Revenue', axis=1)
# Split data into training and testing set
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)<jupyter_output><empty_output><jupyter_text>As usual, it is necessary to oversample the minority class in order to improve the model performance.<jupyter_code># Oversample the minority class in the target variable
oversample = SMOTE()
X_train, y_train = oversample.fit_resample(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Model Training<jupyter_code># Declare estimator
estimator = XGBClassifier(tree_method='gpu_hist', gpu_id='0')

# Declare cross validation method
cv = StratifiedKFold()

# Declare paramter grid
param_grid = dict(
    n_estimators = [50, 100, 200, 400],
    max_depth = [3, 6, 9],
    learning_rate = [1, 0.1, 0.01],
    subsample = [0.5, 0.8, 1],
    colsample_bytree = [0.5, 0.8, 1]
)
# Import grid search from tune sklearn
from tune_sklearn import TuneGridSearchCV

# Train the model
xgb_clf = TuneGridSearchCV(estimator=estimator, param_grid=param_grid, scoring="f1", cv=cv, n_jobs=-1, use_gpu=True, verbose=2)
xgb_clf.fit(X_train, y_train)
xgb_clf.best_estimator_
# Save and load the model if required
import joblib

joblib.dump(xgb_clf.best_estimator_, open('/content/drive/MyDrive/Colab Notebooks/uci/ospi/models/xgb.pkl', 'wb'))
# xgb_clf_loaded = joblib.load(open('/content/drive/MyDrive/Colab Notebooks/uci/ospi/models/xgb.pkl', 'rb'))
# Get predictions from the model
y_pred = xgb_clf.best_estimator_.predict(X_test, validate_features=False)<jupyter_output><empty_output><jupyter_text>Model Evaluation<jupyter_code>print(classification_report(y_test, y_pred))<jupyter_output>precision    recall  f1-score   support

       False       0.95      0.92      0.93      2594
        True       0.63      0.76      0.69       489

    accuracy                           0.89      3083
   macro avg       0.79      0.84      0.81      3083
weighted avg       0.90      0.89      0.90      3083<jupyter_text>Xgboost classifier classifier is as good as the adaboost classifier and is also comparable to the models used in the original paper.  Model Interpretation Xgboost also uses white-box models which helps to explain the model in a better way. The feature importances provided by the model are of great use for understanding how the model has learnt.<jupyter_code># Create a feature importance dataframe
feat_imp_data = zip(list(df.drop('Revenue', axis=1).columns), xgb_clf.best_estimator_.feature_importances_)
feat_imp_df = pd.DataFrame(columns=['column', 'feature_importance'], data=feat_imp_data)
# Sort feature importance
feat_imp_df.sort_values('feature_importance', ascending=False, inplace=True)
fig = px.bar(feat_imp_df[:20], x='feature_importance', y='column', orientation='h')
fig.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08861386138613861, 'fraction_numerical': 0.024504950495049506, 'mean_word_length': 5.058470764617691, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '468836', 'license': 'MIT', 'n_tokens_mistral': 1389, 'n_tokens_neox': 1304, 'n_words': 372, 'path': 'ospi/docs/models/xgboost_train.ipynb', 'repo_name': 'patilpushkarp/online-shoppers-purchasing-intention', 'size': '4040'}