<jupyter_start><jupyter_text>Automating Hyperparameter TuningIn the previous chapter, we manually iterated over different combinations of hyperparameters to find the single final ('best') model that will be used in the future to make predictions on unseen data. In this chapter, we will select this final model by automating the process of hyperparameter selection through a procedure called grid search with help from the `GridSearchCV` meta-estimator. What is a meta-estimator?`GridSearchCV` is not a machine learning model by itself, though it is a scikit-learn estimator and follows the same three-step process. Because it takes an estimator as input during instantiation, it is referred to as a meta-estimator. Let's get started by importing our data and selecting the same features as input as the previous chapter.<jupyter_code>import pandas as pd
%matplotlib inline
housing = pd.read_csv('../data/housing_sample.csv')
X = housing[['GrLivArea', 'GarageArea', 'FullBath']]
y = housing['SalePrice']
X.head()<jupyter_output><empty_output><jupyter_text>The `GridSearchCV` meta-estimatorThe primary focus of this chapter will be on the `GridSearchCV` meta-estimator located in the `model_selection` module. It completes nearly the exact same process as we did in the previous chapter. It iterates over all the combinations of a given sequence of hyperparameters, computes a cross-validated mean score, and then fits all of the data on the best combination of hyperparameters. Create a gridTo use `GridSearchCV`, you must create a dictionary that maps the hyperparameter you want to search as a string name to a sequence of the possible values it can be. Here, we will perform a grid search on a decision tree and seek to optimize the `max_depth` parameter. We will search values between 1 and 20. We begin by creating a dictionary, mapping the string 'max_depth' to the sequence of integers from 1 to 20.<jupyter_code>grid = {'max_depth': range(1, 21)}<jupyter_output><empty_output><jupyter_text>Import estimator and `GridSearchCV` and instantiate themThe `GridSearchCV` meta-estimator must be instantiated with the machine learning estimator along with the grid containing the possible hyperparameter values. The machine learning estimator that we use must also be instantiated. The 'CV' in `GridSearchCV` stands for cross-validation and may be passed a splitter to its `cv` parameter during instantiation. The `GridSearchCV` takes care of the work provided by `cross_val_score` from the previous chapter. In order to make valid comparisons of the results of `GridSearchCV` to the previous chapter, we use the same value for `random_state`.Let's import and instantiate our estimator, meta-estimator, and splitter. The first two parameters to `GridSearchCV` are the estimator and the grid.<jupyter_code>from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import GridSearchCV, KFold
dtr = DecisionTreeRegressor()
kf = KFold(n_splits=5, shuffle=True, random_state=123)
gs = GridSearchCV(dtr, grid, cv=kf)<jupyter_output><empty_output><jupyter_text>Execute the exhaustive grid searchWe are now ready to perform an exhaustive grid search with the help of `GridSearchCV`. We just imported and instantiated it. We just need to fit it with the data. Every single combination of hyperparameter values supplied by the grid will be cross-validated when the `fit` method is called. In this particular search, there is only a single hyperparameter searched, but in general, any number of hyperparameters may be searched.<jupyter_code>gs.fit(X, y);<jupyter_output><empty_output><jupyter_text>Results from the grid search objectOur variable name `gs` contains lots of information on the results of the grid search. The best hyperparmeters are stored in the `best_params_` attribute.<jupyter_code>gs.best_params_<jupyter_output><empty_output><jupyter_text>The best score is found in the `best_score_` attribute.<jupyter_code>gs.best_score_<jupyter_output><empty_output><jupyter_text>These results are the same as those calculated in the previous chapter.  All results are stored in `cv_results_` attributeThe `cv_results_` attribute is a dictionary containing the results from the grid search. It contains the scores of each cross validated section along with the time it took to fit and score the model. We can convert it to a pandas DataFrame to make it easier to read.<jupyter_code>df_results = pd.DataFrame(gs.cv_results_)
df_results.head()<jupyter_output><empty_output><jupyter_text>Let's narrow our focus to just the hyperparameter value and the mean and standard deviation of the cross-validated scores.<jupyter_code>cols = ['param_max_depth', 'mean_test_score', 'std_test_score']
df_results[cols].sort_values('mean_test_score', ascending=False).head(5).round(3)<jupyter_output><empty_output><jupyter_text>We can plot the results just like we did in the preceding chapter.<jupyter_code>df_results.plot(x='param_max_depth', y='mean_test_score', kind='line', figsize=(10, 4));<jupyter_output><empty_output><jupyter_text>Getting the best trained estimator backIn the preceding chapter, we had to train the model on all of the data with the value of the best hyperparameter. With `GridSearchCV` this is done for us. The result is stored in the `best_estimator_` attribute. The best estimator that gets returned has been fit on **all** of the training data. Let's assign this estimator to its own variable name and output the hyperparameters to verify that the `max_depth` has been set correctly.<jupyter_code>dtr_best = gs.best_estimator_
dtr_best.get_params()<jupyter_output><empty_output><jupyter_text>Grid searching multiple hyperparametersExpanding the grid search to more hyperparameters is not difficult. We simply add a new entry in our grid dictionary. Again, the hyperparameter name as a string is used as the key in the dictionary and is mapped to a sequence of all the possible values you want to search. Let's search both `max_depth` and `min_samples_leaf` simultaneously.<jupyter_code>grid = {'max_depth': range(2, 10), 
        'min_samples_leaf': [2, 5, 10, 20, 50, 100]}
gs = GridSearchCV(dtr, grid, cv=kf)
gs.fit(X, y);<jupyter_output><empty_output><jupyter_text>Let's retrieve the best combination of hyperparameters and the resulting best score.<jupyter_code>gs.best_params_
gs.best_score_<jupyter_output><empty_output><jupyter_text>With multiple hyperparameters it may be easier to view the results as a heatmap. The two parameter columns are pivoted and then colored based on their score.<jupyter_code>df_results = pd.DataFrame(gs.cv_results_)
df_results.pivot(index='param_max_depth', 
                 columns='param_min_samples_leaf', 
                 values='mean_test_score').round(3) \
          .style.background_gradient('coolwarm', axis=None)<jupyter_output><empty_output><jupyter_text>Let's continue our automated tuning and use a total of four different hyperparameters. The `criterion` is the error used to determine the best split. By default, it uses the mean squared error, but you also have the choice of using mean absolute error. Each is referenced by the strings 'mse' and 'mae' respectively. The `max_leaf_nodes` limits trees so that the total number of leaves do not exceed the given amount. Let's go straight to the visualization of the results.<jupyter_code>grid = {'max_depth': range(4, 9), 
        'min_samples_leaf': [10, 20, 50],
        'criterion': ['mse', 'mae'],
        'max_leaf_nodes': [10, 20, 30, 50]}

gs = GridSearchCV(dtr, grid, cv=kf)
gs.fit(X, y)
df_results = pd.DataFrame(gs.cv_results_)
df_results.pivot_table(index=['param_max_depth', 'param_criterion'],
                 columns=['param_min_samples_leaf', 'param_max_leaf_nodes'],
                 values='mean_test_score').round(3) \
          .style.background_gradient('coolwarm', axis=None)<jupyter_output><empty_output><jupyter_text>Let's get the best combination of hyperparameters along with the best score directly from `gs`.<jupyter_code>gs.best_params_
gs.best_score_<jupyter_output><empty_output><jupyter_text>Notice how the tuning of these two extra hyperparameters resulted in no increase in the cross-validated score. At some point, no amount of tuning will result in a better cross-validated score. You'll have to use other approaches to make a better model. As always, retrieve the model fitted on all the data with the best hyperparameters.<jupyter_code>dtr_best = gs.best_estimator_<jupyter_output><empty_output><jupyter_text>Grid searching is computationally expensiveGrid searching is doing a tremendous amount of work. It searches all combinations of the hyperparameter space that you supply it with. This type of searching that attempts every combination is known as an **exhaustive grid search**. The search area grows quickly with the addition of each new hyperparameter. If you have 10 choices of values for each of 3 different hyperparameters, the grid search will run 1,000 different models ($10 ^ 3$). Additionally, the grid search is doing cross validation, so if you are using 5 folds, then this turns into 5,000 different trained models.Even on a small dataset like our housing data which has less than 1,500 observations, our last grid search on the four hyperparameters took some time. If we searched a larger area, then its quite possible that our grid search on this small dataset would have taken multiple hours to complete. Parallelizing with `n_jobs` There exists a parameter in `GridSearchCV` named `n_jobs` which allows you to run the grid searching in parallel on multiple CPUs on your machine. By default, scikit-learn uses 1 CPU. Set the number of CPUs with an integer. You can set it to -1 to use all the CPUs on your machine. Let's time the results of grid searching using the defaults and with all the CPUs on my machine.<jupyter_code>import time
start = time.time()
gs = GridSearchCV(dtr, grid, cv=kf)
gs.fit(X, y)
time.time() - start
start = time.time()
gs = GridSearchCV(dtr, grid, cv=kf, n_jobs=-1)
gs.fit(X, y)
time.time() - start<jupyter_output><empty_output><jupyter_text>Typically, you only need to set `n_jobs` whenever you have a large grid to search, otherwise there won't be much, if any, performance benefit from using it.  Reduce computation time with `RandomizedSearchCV`Exhaustive grid searches attempt every single combination, but can be extremely time consuming if you are searching many thousands or even millions of combinations. You can easily build a grid that takes days to completely search. When doing an exhaustive grid search, many of the combinations will be highly suboptimal and worthless to check.Instead of checking all of the combinations of the grid, you can randomly choose a subset of them with `RandomizedSearchCV`. This meta-estimator works almost identically as `GridSearchCV` but has the parameter `n_iter` that you set to an integer which determines the number of random choices to make.Let's take a look at the last grid we used with four hyperparameters. There are 5 choices for `max_depth`, 3 for `min_samples_leaf`, 2 for `criterion`, and 4 for `max_leaf_nodes` making for a total of `5 * 3 * 2 * 4` or 120 total combinations.<jupyter_code>grid<jupyter_output><empty_output><jupyter_text>Let's use `RandomizedSearchCV` to randomly cross-validate 15 of our combinations from the grid. The `random_state` parameter is set so that the same combinations are chosen each time for this book. When you actually use `RandomizedSearchCV` you might not want to set `random_state` so that you get a different set of combinations each time.<jupyter_code>from sklearn.model_selection import RandomizedSearchCV
rs = RandomizedSearchCV(dtr, grid, n_iter=15, cv=kf, random_state=456, n_jobs=-1)
rs.fit(X, y);<jupyter_output><empty_output><jupyter_text>The `RandomizedSearchCV` object contains most of the same attributes as `GridSearchCV`. Let's get the best combination of the 15 parameter combinations it chose and the score for that combination.<jupyter_code>rs.best_params_
rs.best_score_<jupyter_output><empty_output><jupyter_text>This best combination is different than the one found with the exhaustive grid search, though the score is only just slightly less. It would be interesting to know what the rank of this combination would be with respect to all 120 combinations. There is a column in `df_results` named `params` that contains a dictionary of the hyperparameter values. We can create a filter by testing equality using the dictionary stored in `best_params_`. Below, we return columns for the rank, mean test score and value of the hyperparameters.<jupyter_code>filt = df_results['params'] == rs.best_params_
cols = ['params', 'mean_test_score', 'rank_test_score']
df_results.loc[filt, cols]<jupyter_output><empty_output><jupyter_text>This combination ranked fifth out of 120. Although there were several combinations that received a higher score, the score was not that much higher and our model trained in far less time. Exhaustive grid search around a hot spotWhile the `RandomizedSearchCV` might not find the combination that yields the absolute highest cross-validated score, it often provides good choices for the hyperparameters. You can consider using the results of `RandomizedSearchCV` to find 'hot' spots in the grid. For instance, take a look at the `best_params_` dictionary from our last search. We can perform an exhaustive grid search around this one combination. We can even limit our search to just one or two hyperparameters while keeping the others set to one specific value.Let's instantiate a new decision tree, but this time set the hyperparameters `criterion` to 'mse' and `min_samples_leaf` to 20. They will not be searched. Then create a grid for just `max_depth` and `max_leaf_nodes` and perform the exhaustive grid search on the 25 total combinations.<jupyter_code>dtr = DecisionTreeRegressor(criterion='mse', min_samples_leaf=20)
grid = {'max_depth': [5, 6, 7, 8, 9], 
        'max_leaf_nodes': [10, 15, 20, 25, 30]}
gs = GridSearchCV(dtr, grid, cv=kf, n_jobs=-1)
gs.fit(X, y);<jupyter_output><empty_output><jupyter_text>Let's see our new best choices for hyperparameters and their score.<jupyter_code>gs.best_params_
gs.best_score_<jupyter_output><empty_output><jupyter_text>Our score will never decrease as the best combination from the randomized search is included in this one. At worst, it will stay the same. Let's visualize this to get a better understanding of our search results.<jupyter_code>df_results = pd.DataFrame(gs.cv_results_)
df_results.pivot(index='param_max_depth', 
                 columns='param_max_leaf_nodes', 
                 values='mean_test_score').round(3) \
          .style.background_gradient('coolwarm', axis=None)<jupyter_output><empty_output><jupyter_text>Because the best combination was found on the edge of the grid, we might want to run another search centering it around a `max_depth` of 6 and `max_leaf_nodes` of 30.<jupyter_code>grid = {'max_depth': [5, 6, 7], 
        'max_leaf_nodes': range(25, 36)}
gs = GridSearchCV(dtr, grid, cv=kf, n_jobs=-1)
gs.fit(X, y);
df_results = pd.DataFrame(gs.cv_results_)
df_results.pivot(index='param_max_depth', 
                 columns='param_max_leaf_nodes', 
                 values='mean_test_score').round(3) \
          .style.background_gradient('coolwarm', axis=None)<jupyter_output><empty_output><jupyter_text>No combination improves over the previous result. A `max_depth` of 6 and `max_leaf_nodes` of 30 look to be our final choices for these hyperparameters. Let's do one more exhaustive grid search over `min_samples_split` holding the values for the other hyperparameters constant.<jupyter_code>dtr = DecisionTreeRegressor(criterion='mse', max_depth=6, max_leaf_nodes=30)
grid = {'min_samples_leaf': range(10, 40)}
gs = GridSearchCV(dtr, grid, cv=kf, n_jobs=-1)
gs.fit(X, y);
df_results = pd.DataFrame(gs.cv_results_, columns=['params', 'mean_test_score'])
df_results.sort_values('mean_test_score', ascending=False).head().round(3)<jupyter_output><empty_output><jupyter_text>A `min_samples_leaf` of around 20 appears to be the best choice. We were using this above, but this search provides further evidence that it is a good choice. Finally, let's assign our best model to a variable name of its own and output all the hyperparameters.<jupyter_code>dtr_best = gs.best_estimator_
dtr_best.get_params()<jupyter_output><empty_output><jupyter_text>Using different metrics when grid searchingWe previously learned about different scoring metrics. They are available to use when grid searching by passing in their string name to the `scoring` parameter. Let's return the best model ranking with the mean squared log error.<jupyter_code>grid = {'max_depth': range(4, 9), 
        'min_samples_leaf': [10, 20, 50],
        'criterion': ['mse', 'mae'],
        'max_leaf_nodes': [10, 20, 30, 50, 100]}
dtr = DecisionTreeRegressor()
gs = GridSearchCV(dtr, grid, cv=kf, n_jobs=-1, scoring='neg_mean_squared_log_error')
gs.fit(X, y);<jupyter_output><empty_output><jupyter_text>Let's reveal the best combinations for this metric.<jupyter_code>gs.best_params_
gs.best_score_<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.072282830639986, 'fraction_numerical': 0.009801061781693017, 'mean_word_length': 5.5804222648752395, 'pattern_counts': {'":': 0, '<': 117, '<?xml version=': 0, '>': 117, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '29', 'dir': 'jupyter-structured-clean-dedup', 'id': '397976', 'license': 'MIT', 'n_tokens_mistral': 4999, 'n_tokens_neox': 4734, 'n_words': 2070, 'path': 'jupyter_notebooks/machine_learning/mastering_machine_learning/05. Model Selection/02. Automating Hyperparameter Tuning.ipynb', 'repo_name': 'manual123/Nacho-Jupyter-Notebooks', 'size': '17141'}