<jupyter_start><jupyter_text>Building an ensemble model Outline- **Gather already trained models**   - **Stack trained models together to train an ensemble model.**- **Save the best performing ensemble model**- **Observe and compare all results** Set Parent directory to the system for importing modules<jupyter_code>import os
import sys
import inspect
import pathlib

currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
ROOTDIR = os.path.dirname(currentdir)
print('ROOTDIR', ROOTDIR)
sys.path.insert(0,ROOTDIR)

import json
import numpy  as np
import os 
import pandas as pd

from scripts.preprocess import get_feature_list_from_text_excluding_target
from scripts.model import get_model, get_prediction, save_model, load_saved_model
from scripts.evaluate import get_evaluation_report
from scripts.visualization import precision_recall_curve_comparison
from scripts.visualization import roc_curve_comparison
from scripts.utils import set_seed

seed = 112358

set_seed(seed)
results = {}<jupyter_output>ROOTDIR E:\github_projects\Case-Study-Campaign-Optimization<jupyter_text>Gather Data<jupyter_code>full_data = pd.read_csv(fr"{ROOTDIR}/data/prepared_data_for_modeling.csv")

target = 'next_hour_good_performance'
sample_features = get_feature_list_from_text_excluding_target(fr"{ROOTDIR}/data/feature_lists_for_modeling.txt",
                                                              target) 
                                                          
training_data = full_data[full_data['first_24_hour']][sample_features+[target]]                                              
X_train = np.ascontiguousarray(training_data[sample_features])
y_train = np.ascontiguousarray(training_data[target])

test_data = full_data[full_data['test_set']][sample_features+[target]]
X_test = np.ascontiguousarray(test_data[sample_features])
y_test = np.ascontiguousarray(test_data[target])

del training_data, test_data<jupyter_output>Successfully returned list from:  E:\github_projects\Case-Study-Campaign-Optimization/data/feature_lists_for_modeling.txt<jupyter_text>Load Trained Models<jupyter_code>xgb_model = load_saved_model(fr"{ROOTDIR}/trained_models", 'xgb_model')
lgb_model = load_saved_model(fr"{ROOTDIR}/trained_models", 'lightGBM_model')
rf_model = load_saved_model(fr"{ROOTDIR}/trained_models", 'rf_model')
model_results = pd.read_csv(fr"{ROOTDIR}/results/model_results.csv")
model_results.sort_values('pr_auc', ascending = False)<jupyter_output><empty_output><jupyter_text>We can see xgb is the best performing with AUC of 0.26  Ensemble Modeling or voting ClassifierStack already trained models together to train an ensemble model. **XGB and LGB**<jupyter_code>ensemble_classifier = get_model(name='ensemble',
                                estimators=[('lgb_model', lgb_model),('xgb_model',xgb_model)], 
                                voting='soft', 
                                weights=[1,1])
ensemble_classifier.fit(X_train, y_train)

preidiction = get_prediction(ensemble_classifier, X_test)

labels = ['Bad Ads', 'Good Ads']
report = get_evaluation_report(y_test, preidiction, labels, plot='precision-recall')

results['ensemble_xgb_+_lgb'] = [report['auc'],\
                  report['classification_report']['macro avg']['precision'],\
                  report['classification_report']['macro avg']['recall'],\
                  report['classification_report']['macro avg']['f1-score'],\
                  report['mcc']]<jupyter_output>[00:39:42] WARNING: ..\src\learner.cc:1095: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.
PR-curve threshold=0.861960, F-Score=0.369<jupyter_text>**Observation:** Performance increased slightly with compared to xgb but very less 0.001. Let's give more weight to xgb model and see if performance increases<jupyter_code>ensemble_classifier = get_model(name='ensemble',
                                estimators=[('lgb_model', lgb_model),('xgb_model',xgb_model)], 
                                voting='soft', weights=[1,2])
ensemble_classifier.fit(X_train, y_train)

preidiction = get_prediction(ensemble_classifier, X_test)

labels = ['Bad Ads', 'Good Ads']
report = get_evaluation_report(y_test, preidiction, labels, plot='precision-recall')
results['ensemble_weighted_xgb_+_lgb'] = [report['auc'],\
                  report['classification_report']['macro avg']['precision'],\
                  report['classification_report']['macro avg']['recall'],\
                  report['classification_report']['macro avg']['f1-score'],\
                  report['mcc']]<jupyter_output>[00:39:54] WARNING: ..\src\learner.cc:1095: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.
PR-curve threshold=0.853811, F-Score=0.372<jupyter_text>We can observe a better improvement an increase of 0.003 of PR-AUC w.r.t XgBoost **XGB and Random Forest**<jupyter_code>ensemble_classifier = get_model(name='ensemble',estimators=[('xgb', xgb_model),('rf',rf_model)], 
                                voting='soft', 
                                weights=[1,1])
ensemble_classifier.fit(X_train, y_train)

preidiction = get_prediction(ensemble_classifier, X_test)

labels = ['Bad Ads', 'Good Ads']
report = get_evaluation_report(y_test, preidiction, labels, plot='precision-recall')
results['ensemble_xgb_+_rf'] = [report['auc'],\
                  report['classification_report']['macro avg']['precision'],\
                  report['classification_report']['macro avg']['recall'],\
                  report['classification_report']['macro avg']['f1-score'],\
                  report['mcc']]<jupyter_output>[00:40:06] WARNING: ..\src\learner.cc:1095: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.
PR-curve threshold=0.788638, F-Score=0.362<jupyter_text>Giving double weight to XGB model<jupyter_code>ensemble_classifier = get_model(name='ensemble',
                                estimators=[('xgb', xgb_model),('rf',rf_model)], 
                                voting='soft', weights=[2,1])

ensemble_classifier.fit(X_train, y_train)

preidiction = get_prediction(ensemble_classifier, X_test)

labels = ['Bad Ads', 'Good Ads']
report = get_evaluation_report(y_test, preidiction, labels, plot='precision-recall')

results['ensemble_weighted_xgb_+_rf'] = [report['auc'],\
                  report['classification_report']['macro avg']['precision'],\
                  report['classification_report']['macro avg']['recall'],\
                  report['classification_report']['macro avg']['f1-score'],\
                  report['mcc']]<jupyter_output>[00:40:28] WARNING: ..\src\learner.cc:1095: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.
PR-curve threshold=0.803220, F-Score=0.379<jupyter_text>Observation: Nothing Interesting so far **LGB and RF**<jupyter_code>ensemble_classifier = get_model(name='ensemble',
                                estimators=[('lgb',lgb_model),('rf_model',rf_model)], 
                                voting='soft', 
                                weights=[1,1])

ensemble_classifier.fit(X_train, y_train)

preidiction = get_prediction(ensemble_classifier, X_test)

labels = ['Bad Ads', 'Good Ads']
report = get_evaluation_report(y_test, preidiction, labels, plot='precision-recall')


results['ensemble_lgb_rf'] = [report['auc'],\
                  report['classification_report']['macro avg']['precision'],\
                  report['classification_report']['macro avg']['recall'],\
                  report['classification_report']['macro avg']['f1-score'],\
                  report['mcc']]<jupyter_output>PR-curve threshold=0.799571, F-Score=0.352<jupyter_text>**Ensemble of all 3 classifiers**<jupyter_code>ensemble_classifier = get_model(name='ensemble',
                                estimators=[('xgb',xgb_model),
                                            ('lgb',lgb_model),('rf',rf_model)], 
                                            voting='soft', 
                                            weights=[1,1,1])
ensemble_classifier.fit(X_train, y_train)
preidiction = get_prediction(ensemble_classifier, X_test)

labels = ['Bad Ads', 'Good Ads']
report = get_evaluation_report(y_test, preidiction, labels, plot='precision-recall')
results['ensemble_all'] = [report['auc'],\
                  report['classification_report']['macro avg']['precision'],\
                  report['classification_report']['macro avg']['recall'],\
                  report['classification_report']['macro avg']['f1-score'],\
                  report['mcc']]<jupyter_output>[00:41:10] WARNING: ..\src\learner.cc:1095: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.
PR-curve threshold=0.817394, F-Score=0.367<jupyter_text>Giving more weight to XGB model<jupyter_code>ensemble_classifier = get_model(name='ensemble',
                                estimators=[('xgb',xgb_model),('lgb',lgb_model),('rf_model',rf_model)], 
                                voting='soft', 
                                weights=[2,1,1])
ensemble_classifier.fit(X_train, y_train)
preidiction = get_prediction(ensemble_classifier, X_test)

labels = ['Bad Ads', 'Good Ads']
report = get_evaluation_report(y_test, preidiction, labels, plot='precision-recall')

results['ensemble_all_xgb_weighted'] = [report['auc'],\
                  report['classification_report']['macro avg']['precision'],\
                  report['classification_report']['macro avg']['recall'],\
                  report['classification_report']['macro avg']['f1-score'],\
                  report['mcc']]<jupyter_output>[00:41:32] WARNING: ..\src\learner.cc:1095: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.
PR-curve threshold=0.819581, F-Score=0.369<jupyter_text>Observation: We can observe ensemble model of weighted xgb and lightgbm is the best performing amongst all  Gather Result<jupyter_code>ensemble_results = pd.DataFrame.from_dict(results).T
ensemble_results.columns = ['pr_auc', 'macro_precision', 'macro_recall', 'macro_f1', 'mcc']
ensemble_results.to_csv(fr"{ROOTDIR}/results/ensemble_model_results.csv")
ensemble_results.sort_values('pr_auc', ascending=False)<jupyter_output><empty_output><jupyter_text>Save the best ensemble model<jupyter_code>save_path = fr"{ROOTDIR}\\plots\\EnsembleModel\\"

ensemble_model = get_model(name='ensemble',
                                estimators=[('lgb_model', lgb_model),('xgb_model',xgb_model)], 
                                voting='soft', weights=[1,2])

ensemble_model.fit(X_train, y_train)

preidiction = get_prediction(ensemble_model, X_test)

labels = ['Bad Ads', 'Good Ads']
report = get_evaluation_report(y_test, 
                               preidiction, 
                               labels, 
                               plot='precision-recall',
                               save_path = save_path = save_path+'test'
                              )

path = fr"{ROOTDIR}/trained_models"
model_name = 'ensemble_model'
save_model(ensemble_model, path, model_name)<jupyter_output>[02:31:07] WARNING: ..\src\learner.cc:1095: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.
PR-curve threshold=0.853811, F-Score=0.372<jupyter_text>On Training set<jupyter_code>##used to load saved model
#ensemble_trained = load_saved_model(fr"{ROOTDIR}/trained_models", 'ensemble_model')

preidiction = get_prediction(ensemble_model, X_train)
labels = ['Bad Ads', 'Good Ads']
report = get_evaluation_report(y_train, 
                               preidiction, 
                               labels, 
                               plot='precision-recall',
                               save_path = save_path+'train',
                               threshold = report['threshold'],
                              )<jupyter_output>ensemble_model found at E:\Data Science Masters\university\coding challenge\future_demand/model\ensemble_model.
PR-curve threshold=0.832431, F-Score=0.386<jupyter_text>Observe and compare results<jupyter_code>combined_results = model_results.append(ensemble_results)
combined_results.sort_values('pr_auc', ascending=False)<jupyter_output><empty_output><jupyter_text>Let's check everything in a PR-AUC curve (only taking best ensemble model)<jupyter_code>temp_model_dict = {'xgb':xgb_model, 'lgb':lgb_model, 'rf':rf_model, 'ensemble':ensemble_model}
precision_recall_curve_comparison(temp_model_dict, X_test, y_test, save_path = fr"{ROOTDIR}/plots")<jupyter_output><empty_output><jupyter_text>Let's check out everything on Roc Curve<jupyter_code>temp_model_dict = {'xgb':xgb_model, 'lgb':lgb_model, 'rf':rf_model, 'ensemble':ensemble_model}
roc_curve_comparison(temp_model_dict, X_test, y_test, save_path = fr"{ROOTDIR}/plots")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12190142898804317, 'fraction_numerical': 0.01727909011373578, 'mean_word_length': 3.712126417038818, 'pattern_counts': {'":': 0, '<': 54, '<?xml version=': 0, '>': 54, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '267677', 'license': 'MIT', 'n_tokens_mistral': 4495, 'n_tokens_neox': 4196, 'n_words': 815, 'path': 'Modeling/Ensemble Model.ipynb', 'repo_name': 'rashikcs/Case-Study-Campaign-Optimization', 'size': '13716'}