<jupyter_start><jupyter_text>📃 Solution for Exercise M6.04The aim of the exercise is to get familiar with the histogramgradient-boosting in scikit-learn. Besides, we will use this model withina cross-validation framework in order to inspect internal parameters foundvia grid-search.We will use the California housing dataset.<jupyter_code>from sklearn.datasets import fetch_california_housing

data, target = fetch_california_housing(return_X_y=True, as_frame=True)
target *= 100  # rescale the target in k$<jupyter_output><empty_output><jupyter_text>First, create a histogram gradient boosting regressor. You can set thetrees number to be large, and configure the model to use early-stopping.<jupyter_code># solution
from sklearn.ensemble import HistGradientBoostingRegressor

hist_gbdt = HistGradientBoostingRegressor(
    max_iter=1000, early_stopping=True, random_state=0)<jupyter_output><empty_output><jupyter_text>We will use a grid-search to find some optimal parameter for this model.In this grid-search, you should search for the following parameters:* `max_depth: [3, 8]`;* `max_leaf_nodes: [15, 31]`;* `learning_rate: [0.1, 1]`.Feel free to explore the space with additional values. Create thegrid-search providing the previous gradient boosting instance as the model.<jupyter_code># solution
from sklearn.model_selection import GridSearchCV

params = {
    "max_depth": [3, 8],
    "max_leaf_nodes": [15, 31],
    "learning_rate": [0.1, 1],
}

search = GridSearchCV(hist_gbdt, params)<jupyter_output><empty_output><jupyter_text>Finally, we will run our experiment through cross-validation. In this regard,define a 5-fold cross-validation. Besides, be sure to shuffle the data.Subsequently, use the function `sklearn.model_selection.cross_validate`to run the cross-validation. You should also set `return_estimator=True`,so that we can investigate the inner model trained via cross-validation.<jupyter_code># solution
from sklearn.model_selection import cross_validate
from sklearn.model_selection import KFold

cv = KFold(n_splits=5, shuffle=True, random_state=0)
results = cross_validate(
    search, data, target, cv=cv, return_estimator=True, n_jobs=2)<jupyter_output><empty_output><jupyter_text>Now that we got the cross-validation results, print out the mean andstandard deviation score.<jupyter_code># solution
print(f"R2 score with cross-validation:\n"
      f"{results['test_score'].mean():.3f} +/- "
      f"{results['test_score'].std():.3f}")<jupyter_output><empty_output><jupyter_text>Then inspect the `estimator` entry of the results and check the bestparameters values. Besides, check the number of trees used by the model.<jupyter_code># solution
for estimator in results["estimator"]:
    print(estimator.best_params_)
    print(f"# trees: {estimator.best_estimator_.n_iter_}")<jupyter_output><empty_output><jupyter_text>We observe that the parameters are varying. We can get the intuition thatresults of the inner CV are very close for certain set of parameters. Inspect the results of the inner CV for each estimator of the outer CV.Aggregate the mean test score for each parameter combination and make a boxplot of these scores.<jupyter_code># solution
import pandas as pd

index_columns = [f"param_{name}" for name in params.keys()]
columns = index_columns + ["mean_test_score"]

inner_cv_results = []
for cv_idx, estimator in enumerate(results["estimator"]):
    search_cv_results = pd.DataFrame(estimator.cv_results_)
    search_cv_results = search_cv_results[columns].set_index(index_columns)
    search_cv_results = search_cv_results.rename(
        columns={"mean_test_score": f"CV {cv_idx}"})
    inner_cv_results.append(search_cv_results)
inner_cv_results = pd.concat(inner_cv_results, axis=1).T
import matplotlib.pyplot as plt

color = {"whiskers": "black", "medians": "black", "caps": "black"}
inner_cv_results.plot.box(vert=False, color=color)
plt.xlabel("R2 score")
plt.ylabel("Parameters")
_ = plt.title("Inner CV results with parameters\n"
              "(max_depth, max_leaf_nodes, learning_rate)")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0945249202062362, 'fraction_numerical': 0.009329732383992144, 'mean_word_length': 6.288014311270126, 'pattern_counts': {'":': 7, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '141994', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 1293, 'n_tokens_neox': 1250, 'n_words': 377, 'path': 'notebooks/ensemble_sol_04.ipynb', 'repo_name': 'parmentelat/scikit-learn-mooc', 'size': '4073'}