<jupyter_start><jupyter_text>Intro to Random Forests  About this course  Teaching approach This course is being taught by <NAME>, and was developed by Jeremy along with <NAME>. You can find the original notebook and video lectures at [Introduction to Machine Learning for Coders!](http://course18.fast.ai/ml)  Books People wanting to learn Python 3 as a first programming language, should have to take a look at [Learn Python 3 the Hard Way](https://www.amazon.com/gp/product/0134692888). If you're already a general purpose developer, but with no previous Python background, I recommend this book [Python Tricks: A Buffet of Awesome Python Features](https://www.amazon.es/Python-Tricks-Buffet-Awesome-Features/dp/1775093301). The more familiarity you have with numeric programming in Python, the better. If you're looking to improve in this area, we strongly suggest <NAME>'s [Python for Data Analysis, 2nd ed](https://www.amazon.com/Python-Data-Analysis-Wrangling-IPython/dp/1491957662/ref=asap_bc?ie=UTF8).For machine learning with Python, we also recommend these books:- [Introduction to Machine Learning with Python](https://www.amazon.com/Introduction-Machine-Learning-Andreas-Mueller/dp/1449369413): From one of the scikit-learn authors, which is the main library we'll be using- [Python Machine Learning: Machine Learning and Deep Learning with Python, scikit-learn, and TensorFlow, 2nd Edition](https://www.amazon.com/Python-Machine-Learning-scikit-learn-TensorFlow/dp/1787125939/ref=dp_ob_title_bk): New version of a very successful book. A lot of the new material however covers deep learning in Tensorflow, which isn't relevant to this course- [Hands-On Machine Learning with Scikit-Learn and TensorFlow](https://www.amazon.com/Hands-Machine-Learning-Scikit-Learn-TensorFlow/dp/1491962291/ref=pd_lpo_sbs_14_t_0?_encoding=UTF8&psc=1&refRID=MBV2QMFH3EZ6B3YBY40K)and some videos:- [Practical Deep Learning for Coders, v3](https://course.fast.ai/videos/?lesson=1) Hands on deep learning course based on pytorch and fast.ai- [Full Stack Deep Learning](https://fullstackdeeplearning.com/march2019) Guide from start to end on how to develop a machine learning projectFinally, I recommend [An Introduction to Statistical Learning](http://www-bcf.usc.edu/~gareth/ISL/ISLR%20Sixth%20Printing.pdf) (free ebook) for a math & statistics approach to Machine Learning  Imports<jupyter_code>%load_ext autoreload
%autoreload 2
%matplotlib inline
import sys, datetime
sys.path.insert(0,'../src')
from utils import *

from sklearn.ensemble import RandomForestRegressor
from sklearn import metrics
from pathlib import Path
import seaborn as sns
PATH_data = Path('..') / 'data' 
!ls {PATH_data}
!dir {PATH_data / 'raw'}<jupyter_output><empty_output><jupyter_text>Introduction to *Blue Book for Bulldozers*  About...  ...this dataset We will be looking at the [Blue Book for Bulldozers](https://www.kaggle.com/c/bluebook-for-bulldozers) Kaggle Competition: "The goal of the contest is to predict the sale price of a particular piece of heavy equiment at auction based on it's usage, equipment type, and configuration.  The data is sourced from auction result postings and includes information on usage and equipment configurations."This is a very common type of dataset and prediciton problem, and similar to what you may see in your project or workplace.  ...Kaggle Competitions [Kaggle](https://www.kaggle.com/) is an awesome resource for aspiring data scientists or anyone looking to improve their machine learning skills.  There is nothing like being able to get hands-on practice and receiving real-time feedback to help you improve your skills.Kaggle provides:1. Interesting data sets2. Feedback on how you're doing3. A leader board to see what's good, what's possible, and what's state-of-art.4. Blog posts by winning contestants share useful tips and techniques.  The data  Look at the data Kaggle provides info about some of the fields of our dataset; on the [Kaggle Data info](https://www.kaggle.com/c/bluebook-for-bulldozers/data) page they say the following:For this competition, you are predicting the sale price of bulldozers sold at auctions. The data for this competition is split into three parts:- **Train.csv** is the training set, which contains data through the end of 2011.- **Valid.csv** is the validation set, which contains data from January 1, 2012 - April 30, 2012. You make predictions on this set throughout the majority of the competition. Your score on this set is used to create the public leaderboard.- **Test.csv** is the test set, which won't be released until the last week of the competition. It contains data from May 1, 2012 - November 2012. Your score on the test set determines your final rank for the competition.The key fields are in train.csv are:- SalesID: the unique identifier of the sale- MachineID: the unique identifier of a machine.  A machine can be sold multiple times- saleprice: what the machine sold for at auction (only provided in train.csv)- saledate: the date of the sale<jupyter_code>%time df_raw = pd.read_csv(PATH_data / 'raw' / 'Train.csv', low_memory=False, parse_dates=["saledate"])<jupyter_output><empty_output><jupyter_text>In any sort of data science work, it's **important to look at your data**, to make sure you understand the format, how it's stored, what type of values it holds, etc. Even if you've read descriptions about your data, the actual data may not be what you expect.<jupyter_code>df_raw.tail().T
#display_all(df_raw.describe(include='all').T)
df_raw.describe(percentiles=[.5]).T<jupyter_output><empty_output><jupyter_text>Sometimes is very useful to slide the data, or apply some transformation to the data. Most of this kind of operations can be done using the `pandas` library. Take a look to the following commands:<jupyter_code># return an slice of columns
df_raw[['SalesID','SalePrice']].head(10)
# it's equivalent to
#df_raw.loc[:,['SalesID','SalePrice']]
# return an slice of rows and columns
df_raw.loc[10:20, ['SalesID','SalePrice']]
# where filtering
df_raw[(df_raw.fiModelDesc=='35NX2') & (df_raw.datasource==132)]
# group by column and aggregation
df_raw[['YearMade','SalesID']].groupby('YearMade').count().head()
df_raw[['YearMade','UsageBand','SalesID']].groupby('YearMade').agg({'SalesID': 'count','UsageBand': ['count','size','nunique']}).head()
df_raw[['YearMade','UsageBand']].groupby('YearMade').agg({'UsageBand': set}).head()<jupyter_output><empty_output><jupyter_text>> Practical Exercise: Calculate: * (min,mean,max) Sale Price by YearMade* age of each vehicle (datetime.date.today().year - YearMade It's important to note what metric is being used for a project. Generally, selecting the metric(s) is an important part of the project setup. However, in this case Kaggle tells us what metric to use: RMSLE (root mean squared log error) between the actual and predicted auction prices. Therefore we take the log of the prices, so that RMSE will give us what we need (in the following section we will explain RMSE)<jupyter_code>_, axes = plt.subplots(1, 2, figsize=(12,6))
sns.distplot(df_raw.SalePrice, ax=axes[0], kde=False, label = 'SalePrice')
sns.distplot(np.log(df_raw.SalePrice),ax=axes[1], kde=False, label = 'log SalePrice');
df_raw.SalePrice = np.log(df_raw.SalePrice)<jupyter_output><empty_output><jupyter_text>Initial processing<jupyter_code>m = RandomForestRegressor(n_jobs=-1, n_estimators=10)
# The following code is supposed to fail due to string values in the input data
m.fit(df_raw.drop('SalePrice', axis=1), df_raw.SalePrice)<jupyter_output><empty_output><jupyter_text>This dataset contains a mix of **continuous** and **categorical** variables.The method `add_datepart` extracts particular date fields from a complete datetime for the purpose of constructing categoricals.  You should always consider this feature extraction step when working with date-time. Without expanding your date-time into these additional fields, you can't capture any trend/cyclical behavior as a function of time at any of these granularities.<jupyter_code>?? add_datepart
add_datepart(df_raw, 'saledate')
df_raw.filter(regex=("sale.*")).head()<jupyter_output><empty_output><jupyter_text>Second, we will inspect the dataframe column types. Most machine learning models require that non-numerical columns should be transformed to numerical. This is another easy transformation we can get using the Categorical type.<jupyter_code>df_raw_columns_by_type = df_raw.columns.to_series().groupby(df_raw.dtypes).groups
df_raw_columns_by_type<jupyter_output><empty_output><jupyter_text>You should always inspect your data for missing values (NA's). In most common machine learning algorithms, the model doesn't accepts this kind of data, so you need to either remove it or transform it.<jupyter_code>display_all(df_raw.isnull().sum().sort_index()/len(df_raw))
sns.countplot (df_raw.UsageBand);
sns.countplot (df_raw.UsageBand.fillna('Empty'));
df_raw.UsageBand.value_counts(dropna=False)<jupyter_output><empty_output><jupyter_text>In order convert a column to Categorical, we can use the `astype('category')` pandas method. The resulting column will contain an index that references the original value of the data. This transformation also takes care of all missing data, which is assigned the category code -1 (we can specify the order to use for categorical variables if we wish)<jupyter_code>usage_band_categories = df_raw.UsageBand.astype('category')
usage_band_categories.cat.categories
sns.countplot (usage_band_categories.cat.codes);<jupyter_output><empty_output><jupyter_text>But let's save this file for now, since it's already in format can we be stored and accessed efficiently (in this case, we choose parquet):<jupyter_code>df_raw.to_parquet(PATH_data / 'processed' / 'train.parquet')<jupyter_output><empty_output><jupyter_text>Pre-processing In the future we can simply read it from this fast format.<jupyter_code>%time df_raw = pd.read_parquet(PATH_data / 'processed' / 'train.parquet')<jupyter_output><empty_output><jupyter_text>We'll replace categories with their numeric codes, handle missing continuous values, and split the dependent variable into a separate variable.<jupyter_code>columns_cat = df_raw.select_dtypes('object').head().columns.values.tolist()
columns_cont = df_raw.columns.symmetric_difference(columns_cat)
for col in columns_cat:
    df_raw[col] = df_raw[col].astype('category').cat.codes + 1
#df_raw.dtypes.to_frame().applymap(str).reset_index().groupby(0).agg({'index':list}).to_dict()['index']
df_raw.columns.to_series().groupby(df_raw.dtypes).groups<jupyter_output><empty_output><jupyter_text>We're still not quite done - for instance we have some missing values, which we can't pass directly to a random forest.<jupyter_code>#display_all(df_raw.isnull().sum()/len(df_raw))
df_raw.columns[df_raw.isnull().sum() > 0].tolist()<jupyter_output><empty_output><jupyter_text>In this case, we replace NA's by the median of the column, but there exists more strategies to deal with NA's ( _imputation methods_ ):- Remove row / column- **Replace by** Zero / Mean / **Median** / Mode- Linear regression- Soft imputation- Add boolean column isNA<jupyter_code>df_raw.fillna(df_raw.median(), inplace=True)
assert (df_raw.isnull().sum() == 0).all()
df_raw.to_parquet(PATH_data / 'interim' / 'train.parquet')<jupyter_output><empty_output><jupyter_text>We now have something we can pass to a random forest!<jupyter_code>m = RandomForestRegressor(n_jobs=-1, n_estimators=10)
%time m.fit(df_raw, df_raw.SalePrice)<jupyter_output><empty_output><jupyter_text>The method `score()` of the model returns the R^2 metric.<jupyter_code>m.score(df_raw, df_raw.SalePrice)<jupyter_output><empty_output><jupyter_text>> Practical Exercise: The score is too good to be true. Search on the internet about the term [**data leakage**](https://www.google.com/search?q=data+leakage) (or if you're too lazy, click on [link for the lazies](https://insidebigdata.com/2014/11/26/ask-data-scientist-data-leakage/)). What is it? Now, review the training code and fix it to avoid the data leakage. We evaluate machine learning models using **scoring metrics**. As mentioned  before, the original Kaggle competition uses RMSLE, but for our purposes we will use another metric called R^2 (aka as R-squared or [coefficient of determination](https://en.wikipedia.org/wiki/Coefficient_of_determination)). This metric measures the percentage of variance explained by the model, so the value of this metric ranges between 0 and 1 (the higher the better). The definition is as follows:$$r^2=1-\frac{RSS}{TSS}=1-\frac{\sum_{i=1}^{n}(y_i-\hat{y}_i)^2}{\sum_{i=1}^{n}(y_i-\bar{y})^2}$$ > Practical Exercise: RMSLE comes from [RMSE](https://en.wikipedia.org/wiki/Root-mean-square_deviation), which is a very common metric, defined below:> $$RMSE = \sqrt{\frac{\sum_{i=1}^{n}(y_i-\hat{y}_i)^2}{n}}$$> Develop a function that accepts a random forest model (m), a dataframe containing the features (X_train) and another dataframe containing the labels (y_train), and returns the RMSE value  validation set When you fix the data leakage, you will get an R^2 score around 0.98 - that's great, right? Well, perhaps not...Possibly **the most important idea** (and one of the differences respect Statistical methods) in Machine Learning  is that of having separate training & validation data sets. As motivation, suppose you don't divide up your data, but instead use all of it.  And suppose you have lots of parameters:The error for the pictured data points is lowest for the model on the far right (the blue curve passes through the red points almost perfectly), yet it's not the best choice.  Why is that?  If you were to gather some new data points, they most likely would not be on that curve in the graph on the right, but would be closer to the curve in the middle graph.This illustrates how using all our data can lead to **overfitting**. A validation set helps diagnose this problem. [Underfitting and Overfitting](https://datascience.stackexchange.com/questions/361/when-is-a-model-underfitted)<jupyter_code>df = df_raw.drop(columns=['SalePrice'])
y = df_raw.SalePrice
?? split_vals
n_valid = 12_000  # same as Kaggle's test set size
n_trn = len(df)-n_valid
raw_train, raw_valid = split_vals(df_raw, n_trn)
X_train, X_valid = split_vals(df, n_trn)
y_train, y_valid = split_vals(y, n_trn)

X_train.shape, y_train.shape, X_valid.shape<jupyter_output><empty_output><jupyter_text>Random Forests  Base model Let's try our model again, this time with separate training and validation sets.<jupyter_code>import pprint

def print_score(m):
    output = {
        'RMSE train' : rmse(m.predict(X_train), y_train),
        'RMSE val'   : rmse(m.predict(X_valid), y_valid),
        'R^2 train'  : m.score(X_train, y_train),
        'R^2 val'    : m.score(X_valid, y_valid)
    }
    if hasattr(m, 'oob_score_'): 
        output['R^2 oob'] = m.oob_score_
    
    pprint.pprint (output)
m = RandomForestRegressor(n_jobs=-1, n_estimators=10)
%time m.fit(X_train, y_train)
print_score(m)<jupyter_output><empty_output><jupyter_text>We are using a helping method named `print_score()` which display main metrics of the model. Besides the R^2 mentioned before, now we display also the RMSE.An R^2 in the high-80's isn't bad at all, but we can see from the validation set score that we're over-fitting badly. To understand this issue, let's simplify things down to a single small tree.  Speeding things up<jupyter_code>df_subsample = df_raw.sample(n=20_000)
X_train = df_subsample.drop(columns=["SalePrice"])
y_train = df_subsample.SalePrice
m = RandomForestRegressor(n_jobs=-1, n_estimators=10)
%time m.fit(X_train, y_train)
print_score(m)<jupyter_output><empty_output><jupyter_text>Before moving on, let's define an **Stochastic model**, which represents a model with some degree of uncertainty, where each event involved in the model has an associated degree of probability. The opposity is a **Deterministic model**, where the output (prediction) is certain. In simple words, this means real life models are stochastic, and by this, some level of randomness is added in the implementation of a machine learning model. As result, executing two times the same training, returns different results.<jupyter_code>m = RandomForestRegressor(n_jobs=-1, n_estimators=10)
%time m.fit(X_train, y_train)
print_score(m)<jupyter_output><empty_output><jupyter_text>We can achieve some degree of reproducibility by setting up a random seed in the frameworks / libraries<jupyter_code>m = RandomForestRegressor(n_jobs=-1, n_estimators=10, random_state=42)
%time m.fit(X_train, y_train)
print_score(m)
m = RandomForestRegressor(n_jobs=-1, n_estimators=10, random_state=42)
%time m.fit(X_train, y_train)
print_score(m)<jupyter_output><empty_output><jupyter_text>Single tree When we disable bootstrap in random forest, we take out the "randomness" in selecting the rows, so the resulting tree takes all available samples (max_sample=1)<jupyter_code>m = RandomForestRegressor(n_estimators=1, max_depth=3, bootstrap=False, n_jobs=-1)
m.fit(X_train, y_train)
print_score(m)<jupyter_output><empty_output><jupyter_text>In order to visualize the following tree, you need to install [GraphViz - Windows Packages](https://graphviz.gitlab.io/_pages/Download/Download_windows.html)<jupyter_code>draw_tree(m.estimators_[0], X_train, precision=3)<jupyter_output><empty_output><jupyter_text>Here’s an understanding of tree and its parameters.1. Feature name — Feature at every node is decided after selecting a feature from a subset of all features.Feature selection based on Mean Squared Error (${\frac{\sum_{i=1}^{n}(y_i-\hat{y}_i)^2}{n}}$) or gini (or information gain) (gini is commonly used because it is less computational complex).2. Split value — split value is decided after selecting a threshold value which gives highest MSE (regression) or gini (classification) for that split3. MSE (or gini) — It is basically deciding factor i.e. to select feature at next node , to pick best split value etc.4. Samples — Number of samples remaining at that particular node.5. Value — Mean value of the samples at this node. In case of regression, is the average of the labels under this tree / leaf. In case of classication, number of samples of each class remaining at that particular node (theoretically sum of these values equals Samples value) The learning curve is another tool commonly used to represent the fitness of the model. Usually plots the improvement of the model (accuracy / loss function) respect training time / number of samples / epochs or any other kind of measure that changes by time.<jupyter_code>plot_learning_curve(m, X_train, y_train)<jupyter_output><empty_output><jupyter_text>Let's see what happens if we create a bigger tree. With enogh depth in the tree, and with no bootstrapping at all, the random forest is able to memorize the whole dataset (overfitting the training set)<jupyter_code>#max_depth:None = nodes are expanded until all leaves are pure or until all leaves contain less than min_samples_split samples
m = RandomForestRegressor(n_estimators=1, max_depth=None, bootstrap=False, n_jobs=-1)
m.fit(X_train, y_train)
print_score(m)<jupyter_output><empty_output><jupyter_text>The training set result looks great! But the validation set is worse than our original model. This is why we need to use *bagging* of multiple trees to get more generalizable results.<jupyter_code>plot_learning_curve(m, X_train, y_train)<jupyter_output><empty_output><jupyter_text>Bagging  Intro to bagging Random Forest = Bag of Little BootstrapsTo learn about bagging in random forests, let's start with our basic model again.<jupyter_code>m = RandomForestRegressor(n_jobs=-1, n_estimators=10)
%time m.fit(X_train, y_train)
print_score(m)<jupyter_output><empty_output><jupyter_text>We'll grab the predictions for each individual tree (`estimators_`), and look at one example, taking the average of all trees and compare with the actual value<jupyter_code>preds = np.stack([t.predict(X_valid) for t in m.estimators_])
preds.shape
assert preds.shape[0] == len(m.estimators_) == m.n_estimators
assert preds.shape[1] == X_valid.shape[0]
preds[:,0], np.mean(preds[:,0]), m.predict([X_valid.iloc[0]])[0], y_valid.iloc[0]<jupyter_output><empty_output><jupyter_text>As we keep adding trees, the rmse score improves (one tree is not worth it, but all combined trees are worth it)<jupyter_code>learning_curve = pd.DataFrame({'Score (r2)': [metrics.r2_score(y_valid, np.mean(preds[:i+1], axis=0)) for i in range(m.n_estimators)],
                               'n_estimators': range(m.n_estimators)})
sns.lineplot(y='Score (r2)', x='n_estimators', data=learning_curve);<jupyter_output><empty_output><jupyter_text>The curve shows that the model improves with the size of the estimators, but this growing is faster at the beginning, and stalls towards the end. So the question is ... until when is it worth to keep adding estimators? The shape of this curve suggests that adding more trees isn't going to help us much. Let's check. (Compare this to our original model on a sample)<jupyter_code>m = RandomForestRegressor(n_estimators=20, n_jobs=-1)
%time m.fit(X_train, y_train)
print_score(m)
m = RandomForestRegressor(n_estimators=40, n_jobs=-1)
%time m.fit(X_train, y_train)
print_score(m)
m = RandomForestRegressor(n_estimators=80, n_jobs=-1)
%time m.fit(X_train, y_train)
print_score(m)
#plt.plot([metrics.r2_score(y_valid, np.mean(preds[:i+1], axis=0)) for i in range(m.n_estimators)]);
learning_curve = pd.DataFrame({'Score (r2)': [metrics.r2_score(y_valid, np.mean(preds[:i+1], axis=0)) for i in range(m.n_estimators)],
                               'n_estimators': range(m.n_estimators)})
sns.lineplot(y='Score (r2)', x='n_estimators', data=learning_curve, label="learning curve", legend=False)
ax = plt.twinx()
sns.lineplot(x=[10,20,40,80], y=[1,1.5,3,5], color="r", ax=ax, label="training time", legend=False)
ax.figure.legend(loc='lower right');<jupyter_output><empty_output><jupyter_text>Out-of-bag (OOB) score If our validation set worse than our training set because we're over-fitting, or because the validation set is for a different time period, or a bit of both? With the existing information we've shown, we can't tell. However, random forests have a very clever trick called *out-of-bag (OOB) error* which can handle this (and more!)The idea is to calculate error on the training set, but only include the trees in the calculation of a row's error where that row was *not* included in training that tree. This allows us to see whether the model is over-fitting, without needing a separate validation set.This also has the benefit of allowing us to see whether our model generalizes, even if we only have a small amount of data so want to avoid separating some out to create a validation set.This is as simple as adding one more parameter to our model constructor. We print the OOB error last in our `print_score` function below.<jupyter_code>m = RandomForestRegressor(n_estimators=15, n_jobs=-1, oob_score=True)
m.fit(X_train, y_train)
print_score(m)
m = RandomForestRegressor(n_estimators=40, n_jobs=-1, oob_score=True)
m.fit(X_train, y_train)
print_score(m)<jupyter_output><empty_output><jupyter_text>Reducing over-fitting  Subsampling  [`set_rf_samples`] It turns out that one of the easiest ways to avoid over-fitting is also one of the best ways to speed up analysis: *subsampling*. Let's return to using our full dataset, so that we can demonstrate the impact of this technique.<jupyter_code>X_train, X_valid = split_vals(df_raw.drop(columns=['SalePrice']), n_trn)
y_train, y_valid = split_vals(df_raw.SalePrice, n_trn)<jupyter_output><empty_output><jupyter_text>The basic idea is this: rather than limit the total amount of data that our model can access, let's instead limit it to a *different* random subset per tree. That way, given enough trees, the model can still see *all* the data, but for each individual tree it'll be just as fast as if we had cut down our dataset as before.<jupyter_code>set_rf_samples(20000)
m = RandomForestRegressor(n_estimators=10, n_jobs=-1, oob_score=True)
%time m.fit(X_train, y_train)
print_score(m)<jupyter_output><empty_output><jupyter_text>Since each additional tree allows the model to see more data, this approach can make additional trees more useful.<jupyter_code>m = RandomForestRegressor(n_estimators=40, n_jobs=-1, oob_score=True)
m.fit(X_train, y_train)
print_score(m)
#plot_learning_curve(m, X_train, y_train)<jupyter_output><empty_output><jupyter_text>We revert to using a full bootstrap sample in order to show the impact of other over-fitting avoidance methods.<jupyter_code>reset_rf_samples()<jupyter_output><empty_output><jupyter_text>Tree building parameters Let's get a baseline for this full set to compare to.<jupyter_code>m = RandomForestRegressor(n_estimators=40, n_jobs=-1, oob_score=True)
m.fit(X_train, y_train)
print_score(m)<jupyter_output><empty_output><jupyter_text>One factor of complexity is the estimator (tree) depth, so if we reduce it, the model will overfit less<jupyter_code>t=m.estimators_[0].tree_
dectree_max_depth(t)
max([dectree_max_depth(est.tree_) for est in m.estimators_])
m = RandomForestRegressor(n_estimators=40, min_samples_leaf=5, n_jobs=-1, oob_score=True)
m.fit(X_train, y_train)
print_score(m)<jupyter_output><empty_output><jupyter_text>Another way to reduce over-fitting is to grow our trees less deeply. We do this by specifying (with **`min_samples_leaf`**) that we require some minimum number of rows in every leaf node. This has two benefits:- There are less decision rules for each leaf node; simpler models should generalize better- The predictions are made by averaging more rows in the leaf node, resulting in less volatility- recommended values: `[1, 3, 5, 10, 25, 100]`<jupyter_code>min([dectree_max_depth(est.tree_) for est in m.estimators_])
m = RandomForestRegressor(n_estimators=40, min_samples_leaf=3, n_jobs=-1, oob_score=True)
m.fit(X_train, y_train)
print_score(m)
min([dectree_max_depth(est.tree_) for est in m.estimators_])
m = RandomForestRegressor(n_estimators=40, min_samples_leaf=3, max_features=0.5, n_jobs=-1, oob_score=True)
m.fit(X_train, y_train)
print_score(m)<jupyter_output><empty_output><jupyter_text>We can also increase the amount of variation amongst the trees by not only use a sample of rows for each tree, but to also using a sample of *columns* for each *split*. We do this by specifying **`max_features`**, which is the proportion of features to randomly select from at each split.- recommended values: `[None, 0.5, sqrt]`<jupyter_code>min([dectree_max_depth(est.tree_) for est in m.estimators_])
m<jupyter_output><empty_output><jupyter_text>> Practical Exercise: Check if you can get a better model, by changing the values of the hyperparameters `max_features`, `min_samples_leaf` and `n_estimators`. These parameters are well documented (run `? RandomForestRegressor` command), but you can also test using other hyperparameters. The sklearn docs [show an example](http://scikit-learn.org/stable/auto_examples/ensemble/plot_ensemble_oob.html) of different `max_features` methods with increasing numbers of trees - as you see, using a subset of features on each split requires using more trees, but results in better models. The green line represents trees that take all the features, but this approach get worse results: not always pays better to take all the data, but column sampling works much better.![sklearn max_features chart](http://scikit-learn.org/stable/_images/sphx_glr_plot_ensemble_oob_001.png)  Finding best hyperparameters using Grid Search Grid Search is a term related to finding the best hyperparameters. We can preset some values to use, and then grid search tries every combination and calculates best improvements using cross validation.<jupyter_code>from sklearn.model_selection import GridSearchCV
parameters = {'max_features': [None, 0.5, 'sqrt'], 
              'min_samples_leaf': [1, 5, 10, 25, 100], 
              'n_estimators': [10, 25, 50],
              'oob_score': [False] }
gs = GridSearchCV(m, parameters, cv=3)
%time gs.fit(X_train, y_train)
gs.best_estimator_
print_score(gs.best_estimator_)
t=gs.best_estimator_.estimators_[0].tree_
dectree_max_depth(t)
max([dectree_max_depth(est.tree_) for est in gs.best_estimator_.estimators_])
pd.DataFrame([dectree_max_depth(est.tree_) for est in gs.best_estimator_.estimators_]).describe(percentiles=[.5])[0].to_dict()<jupyter_output><empty_output><jupyter_text>Save Best Model<jupyter_code>from joblib import dump, load
PATH_models = Path('../models')
dump(gs.best_estimator_, PATH_models / 'bulldozers.pkl')
m_loaded = load(PATH_models / 'bulldozers.pkl')
m_loaded<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09693345386273555, 'fraction_numerical': 0.012029761490855991, 'mean_word_length': 5.991492464754497, 'pattern_counts': {'":': 0, '<': 172, '<?xml version=': 0, '>': 179, 'https://': 17, 'lorem ipsum': 0, 'www.': 11, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '42', 'dir': 'jupyter-structured-clean-dedup', 'id': '224781', 'license': 'MIT', 'n_tokens_mistral': 9270, 'n_tokens_neox': 8796, 'n_words': 3133, 'path': 'notebooks/ml1.ipynb', 'repo_name': 'franperezlopez/mlcourse', 'size': '28782'}