<jupyter_start><jupyter_text>📝 Exercise M3.01The goal is to write an exhaustive search to find the best parameterscombination maximizing the model generalization performance.Here we use a small subset of the Adult Census dataset to make the codefaster to execute. Once your code works on the small subset, try tochange `train_size` to a larger value (e.g. 0.8 for 80% instead of20%).<jupyter_code>import pandas as pd

from sklearn.model_selection import train_test_split

adult_census = pd.read_csv("../datasets/adult-census.csv")

target_name = "class"
target = adult_census[target_name]
data = adult_census.drop(columns=[target_name, "education-num"])

data_train, data_test, target_train, target_test = train_test_split(
    data, target, train_size=0.8, random_state=42)
from sklearn.compose import ColumnTransformer
from sklearn.compose import make_column_selector as selector
from sklearn.preprocessing import OrdinalEncoder

categorical_preprocessor = OrdinalEncoder(handle_unknown="use_encoded_value",
                                          unknown_value=-1)
preprocessor = ColumnTransformer(
    [('cat_preprocessor', categorical_preprocessor,
      selector(dtype_include=object))],
    remainder='passthrough', sparse_threshold=0)

from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.pipeline import Pipeline

model = Pipeline([
    ("preprocessor", preprocessor),
    ("classifier", HistGradientBoostingClassifier(random_state=42))
])<jupyter_output><empty_output><jupyter_text>Use the previously defined model (called `model`) and using two nested `for`loops, make a search of the best combinations of the `learning_rate` and`max_leaf_nodes` parameters. In this regard, you will need to train and testthe model by setting the parameters. The evaluation of the model should beperformed using `cross_val_score` on the training set. We will use thefollowing parameters search:- `learning_rate` for the values 0.01, 0.1, 1 and 10. This parameter controls  the ability of a new tree to correct the error of the previous sequence of  trees- `max_leaf_nodes` for the values 3, 10, 30. This parameter controls the  depth of each tree.<jupyter_code>from sklearn.model_selection import cross_val_score

for l_rate in [1e-2, 1e-1, 1, 10]:
    for m_l_nodes in [3, 10, 30]:
        model.set_params(
            classifier__learning_rate=l_rate,
            classifier__max_leaf_nodes=m_l_nodes,
        )
        scores = cross_val_score(model, data_train, target_train)

        print(f"Accuracy via cross-validation with learning_rate={l_rate} and max_leaf_nodes={m_l_nodes} :\n"
              f"{scores.mean():.3f} +/- {scores.std():.3f}")<jupyter_output>Accuracy via cross-validation with learning_rate=0.01 and max_leaf_nodes=3 :
0.790 +/- 0.004
Accuracy via cross-validation with learning_rate=0.01 and max_leaf_nodes=10 :
0.814 +/- 0.002
Accuracy via cross-validation with learning_rate=0.01 and max_leaf_nodes=30 :
0.842 +/- 0.007
Accuracy via cross-validation with learning_rate=0.1 and max_leaf_nodes=3 :
0.849 +/- 0.004
Accuracy via cross-validation with learning_rate=0.1 and max_leaf_nodes=10 :
0.863 +/- 0.005
Accuracy via cross-validation with learning_rate=0.1 and max_leaf_nodes=30 :
0.861 +/- 0.006
Accuracy via cross-validation with learning_rate=1 and max_leaf_nodes=3 :
0.854 +/- 0.006
Accuracy via cross-validation with learning_rate=1 and max_leaf_nodes=10 :
0.837 +/- 0.008
Accuracy via cross-validation with learning_rate=1 and max_leaf_nodes=30 :
0.819 +/- 0.013
Accuracy via cross-validation with learning_rate=10 and max_leaf_nodes=3 :
0.288 +/- 0.008
Accuracy via cross-validation with learning_rate=10 and max_leaf_nodes=10 :
0.[...]<jupyter_text>Now use the test set to score the model using the best parametersthat we found using cross-validation in the training set.<jupyter_code>best_params = {'learning-rate': 0.1, 'max leaf nodes': 10}

model.set_params(
            classifier__learning_rate=best_params['learning-rate'],
            classifier__max_leaf_nodes=best_params['max leaf nodes'],
        )

model.fit(data_train, target_train)
test_score = model.score(data_test, target_test)

print(f"Test score after the parameter tuning: {test_score:.3f}")<jupyter_output>Test score after the parameter tuning: 0.877
{'fraction_non_alphanumeric': 0.08861351414542903, 'fraction_numerical': 0.04068272153378536, 'mean_word_length': 4.77327935222672, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '153185', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 1448, 'n_tokens_neox': 1307, 'n_words': 400, 'path': 'notebooks/26 - parameter_tuning_ex_02.ipynb', 'repo_name': 'aquinquenel/scikit-learn-mooc', 'size': '4277'}