<jupyter_start><jupyter_text>🏁 Wrap-up quiz**This quiz requires some programming to be answered.**Open the dataset `blood_transfusion.csv`.<jupyter_code>import pandas as pd

blood_transfusion = pd.read_csv("../datasets/blood_transfusion.csv")
data = blood_transfusion.drop(columns="Class")
target = blood_transfusion["Class"]<jupyter_output><empty_output><jupyter_text>In this dataset, the column `"Class"` is the target vector containing thelabels that our model should predict.For all the questions below, make a cross-validation evaluation using a10-fold cross-validation strategy.Evaluate the performance of a `sklearn.dummy.DummyClassifier` that alwayspredict the most frequent class seen during the training. Be aware that you canpass a list of score to compute in `sklearn.model_selection.cross_validate` bysetting the parameter `scoring`.<jupyter_code>from sklearn.dummy import DummyClassifier
from sklearn.model_selection import cross_validate

dummy = DummyClassifier(strategy="most_frequent")
cv = 10

result_dummy = cross_validate(dummy, data, target, cv=cv, n_jobs=-1, scoring=['accuracy', 'balanced_accuracy'])
# test_score_dummy = pd.Series(result_dummy["test_score"], name="Dummy score")
result_dummy
print(f'Accuracy {100*result_dummy["test_accuracy"].mean():0.02f}% +/- {100*result_dummy["test_accuracy"].std():0.02f}')<jupyter_output>Accuracy 76.21% +/- 0.41<jupyter_text>Question 1What the accuracy of this dummy classifier?- a) ~0.5- b) ~0.62- c) ~0.75_Select a single answer_<jupyter_code>print(f'Balanced Accuracy {100*result_dummy["test_balanced_accuracy"].mean():0.02f}% +/- {100*result_dummy["test_balanced_accuracy"].std():0.02f}')<jupyter_output>Balanced Accuracy 50.00% +/- 0.00<jupyter_text>Question 2What the balanced accuracy of this dummy classifier?- a) ~0.5- b) ~0.62- c) ~0.75_Select a single answer_ Replace the `DummyClassifier` by a `sklearn.tree.DecisionTreeClassifier` andcheck the statistical performance to answer the question below.<jupyter_code>from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_validate

tree = DecisionTreeClassifier()
cv = 10

result_tree = cross_validate(tree, data, target, cv=cv, n_jobs=-1, scoring=['accuracy', 'balanced_accuracy'])
# test_score_dummy = pd.Series(result_dummy["test_score"], name="Dummy score")
result_tree
print(f'Accuracy {100*result_tree["test_accuracy"].mean():0.02f}% +/- {100*result_tree["test_accuracy"].std():0.02f}')<jupyter_output>Accuracy 62.86% +/- 14.98<jupyter_text>Question 3Is a single decision classifier better than a dummy classifier (at least anincrease of 4%) in terms of balanced accuracy?- a) Yes- b) No_Select a single answer_ Evaluate the performance of a `sklearn.ensemble.RandomForestClassifier` using300 trees.<jupyter_code>from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_validate

rf = RandomForestClassifier(n_estimators=300)
cv = 10

result_rf = cross_validate(rf, data, target, cv=cv, n_jobs=-1, scoring=['accuracy', 'balanced_accuracy'])
result_rf
print(f'Accuracy {100*result_rf["test_accuracy"].mean():0.02f}% +/- {100*result_rf["test_accuracy"].std():0.02f}')<jupyter_output>Accuracy 66.46% +/- 13.45<jupyter_text>Question 4Is random forest better than a dummy classifier (at least an increase of 4%)in terms of balanced accuracy?- a) Yes- b) No_Select a single answer_ Compare a `sklearn.ensemble.GradientBoostingClassifier` and a`sklearn.ensemble.RandomForestClassifier` with both 300 trees. Evaluate bothmodels with a 10-fold cross-validation and repeat the experiment 10 times.<jupyter_code>from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import cross_validate
from statistics import mean

rf = RandomForestClassifier(n_estimators=300)
gb = GradientBoostingClassifier(n_estimators=300)
cv = 10

rf_results=[]
gb_results=[]

for i in range(10):
    result_rf = cross_validate(rf, data, target, cv=cv, n_jobs=-1, scoring=['accuracy', 'balanced_accuracy'])
    result_gb = cross_validate(gb, data, target, cv=cv, n_jobs=-1, scoring=['accuracy', 'balanced_accuracy'])
    rf_results.append(100*result_rf["test_accuracy"].mean())
    gb_results.append(100*result_gb["test_accuracy"].mean())

print(f'Moyenne RF: {mean(rf_results)}, moyenne GB: {mean(gb_results)}')<jupyter_output>Moyenne RF: 66.86054054054054, moyenne GB: 69.11981981981982<jupyter_text>Question 5On average, is the gradient boosting better than the random forest?- a) Yes- b) No- c) Equivalent_Select a single answer_ Evaluate the performance of a`sklearn.ensemble.HistGradientBoostingClassifier`. Enable early-stopping andadd as many trees as needed.**Note**: Be aware that you need a specific import when importing the`HistGradientBoostingClassifier`:```py explicitly require this experimental featurefrom sklearn.experimental import enable_hist_gradient_boosting now you can import normally from ensemblefrom sklearn.ensemble import HistGradientBoostingClassifier```<jupyter_code>from sklearn.experimental import enable_hist_gradient_boosting
# now you can import normally from ensemble
from sklearn.ensemble import HistGradientBoostingClassifier
from statistics import mean
from sklearn.model_selection import cross_validate

hgb = HistGradientBoostingClassifier(early_stopping=True, max_iter=1000)
hgb_results=[]

for i in range(10):
    result_hgb = cross_validate(hgb, data, target, cv=cv, n_jobs=-1, scoring=['accuracy', 'balanced_accuracy'])
    hgb_results.append(100*result_hgb["test_accuracy"].mean())

print(f'Moyenne HGB: {mean(hgb_results)}')<jupyter_output>Moyenne HGB: 72.87351351351352<jupyter_text>Question 6Is histogram gradient boosting a better classifier?- a) Histogram gradient boosting is the best estimator- b) Histogram gradient boosting is better than random forest by worse than  the exact gradient boosting- c) Histogram gradient boosting is better than the exact gradient boosting but  worse than the random forest- d) Histogram gradient boosting is the worst estimator_Select a single answer_<jupyter_code>result_hgb = cross_validate(hgb, data, target, cv=cv, n_jobs=-1, scoring=['accuracy', 'balanced_accuracy'], return_estimator=True)
n_iter=[]
for est in result_hgb['estimator']:
    n_iter.append(est.n_iter_)
    
print(f'{mean(n_iter)}')<jupyter_output>28.5<jupyter_text>Question 7With the early stopping activated, how many trees on average the`HistGradientBoostingClassifier` needed to converge?- a) ~30- b) ~100- c) ~150- d) ~200- e) ~300_Select a single answer_ [Imbalanced-learn](https://imbalanced-learn.org/stable/) is an open-sourcelibrary relying on scikit-learn and provides methods to deal withclassification with imbalanced classes.Here, we will be using the class[`imblearn.ensemble.BalancedBaggingClassifier`](https://imbalanced-learn.org/stable/references/generated/imblearn.ensemble.BalancedBaggingClassifier.html)to alleviate the issue of class imbalance.Use the `BalancedBaggingClassifier` and pass an`HistGradientBoostingClassifier` as a `base_estimator`. Fix the hyperparameter`n_estimators` to 50.**Note**: In case that imbalanced-learn is not available on your laptop, youcan install it via PyPI or conda-forge channel. Thus in a notebook, you caninstall using:```%pip install -U imbalanced-learn```or```!conda install imbalannced-learn -c conda-forge```  Question 8What is a [`BalancedBaggingClassifier`](https://imbalanced-learn.org/stable/ensemble.htmlbagging)?- a) Is a classifier that make sure that each tree leaves belong to the same  depth level- b) Is a classifier that explicitly maximizes the balanced accuracy score- c) Equivalent to a `sklearn.ensemble.BaggingClassifier` with a resampling of     each bootstrap sample to contain a many samples from each class._Select a single answer_<jupyter_code>from imblearn.ensemble import BalancedBaggingClassifier

bbc = BalancedBaggingClassifier(base_estimator=HistGradientBoostingClassifier(), sampling_strategy='auto', n_estimators=50)
result_bbc = cross_validate(bbc, data, target, cv=cv, n_jobs=-1, scoring=['accuracy', 'balanced_accuracy'], return_estimator=True)


print(f'Accuracy {100*result_bbc["test_balanced_accuracy"].mean():0.02f}% +/- {100*result_bbc["test_balanced_accuracy"].std():0.02f}')<jupyter_output>Accuracy 58.54% +/- 6.57
{'fraction_non_alphanumeric': 0.10476991469422084, 'fraction_numerical': 0.029556650246305417, 'mean_word_length': 7.233432245301682, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '592005', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 2856, 'n_tokens_neox': 2577, 'n_words': 713, 'path': 'jupyter-book/ensemble/module 6 - wrap-up quizz.ipynb', 'repo_name': 'castorfou/scikit-learn-mooc', 'size': '8323'}