<jupyter_start><jupyter_text>4. Fine tuning our modelSpot checking algorithms helped us identify a candidate to model our issue. In our current projet, it's a random forest algorithm. We will now try to fine tune this type of model.What method is best suited? This article on __[Hyperparameter Tuning the Random Forest in Python](https://towardsdatascience.com/hyperparameter-tuning-the-random-forest-in-python-using-scikit-learn-28d2aa77dd74)__ provides a good methodology that we follow here, as it allows for:1. first quick browsing hyperparameter range across a Random Hyperparameter Grid. This helps narrow down our research for the best hyperparameters. 2. then focus on a smaller grid where all combinations of the specified hyperparameters will be tested. Data preprocessing<jupyter_code># import libraries
import pandas as pd
import numpy as np
# load data
path = " "
df = pd.read_excel(path+"Real estate valuation data set.xlsx")
df.info()

df.set_index('No', inplace = True)

# Define the variables
X = df.drop(['Y house price of unit area','X6 longitude'], axis =1)
y = df['Y house price of unit area'].values
# Splitting the dataset into the Training set and Test set
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 0)
# Normalise and scale the numerical features
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import MinMaxScaler
ss_X = StandardScaler()
X_train = ss_X.fit_transform(X_train)
mms_X = MinMaxScaler(feature_range=(0, 1))
X_train = mms_X.fit_transform(X_train)
# Normalise and scale the numerical features - X_test
X_test = ss_X.transform(X_test)
X_test = mms_X.transform(X_test)<jupyter_output>//anaconda/lib/python3.5/site-packages/ipykernel/__main__.py:2: DataConversionWarning: Data with input dtype int64, float64 were all converted to float64 by StandardScaler.
  from ipykernel import kernelapp as app<jupyter_text>BaselineFirst, we look at the default parameters of our random forest model and its performance. We will use it as our baseline.<jupyter_code>from sklearn.ensemble import RandomForestRegressor
regressor = RandomForestRegressor()

regressor.fit(X_train, y_train)

from pprint import pprint
# Look at parameters used by our current forest
print('Parameters currently in use:\n')
pprint(regressor.get_params())
# Predicting the Test set results
y_pred = regressor.predict(X_test)
# Define a function for model evaluation using cross validation
def evaluate_model_cross_validation(name, model, X_train, y_train, folds = 10):

    from sklearn.model_selection import cross_val_score
 
    # Cross Validation Regression MAE
    metric='neg_mean_absolute_error'
    scores = cross_val_score(regressor, X_train, y_train, scoring=metric, cv=folds, n_jobs=-1)
    mean_score, std_score = np.mean(scores), np.std(scores)
    print('>%s - training - MAE: %.3f (+/-%.3f)' % (name, mean_score, std_score))
    
    
    # Cross Validation Regression MSE
    metric='neg_mean_squared_error'
    scores = cross_val_score(regressor, X_train, y_train, scoring=metric, cv=folds, n_jobs=-1)
    mean_score, std_score = np.mean(scores), np.std(scores)
    print('>%s - training - MSE: %.3f (+/-%.3f)' % (name, mean_score, std_score))
    
    # Cross Validation Regression R^2
    metric='r2'
    scores = cross_val_score(regressor, X_train, y_train, scoring=metric, cv=folds, n_jobs=-1)
    mean_score, std_score = np.mean(scores), np.std(scores)
    print('>%s - training - R^2: %.3f (+/-%.3f)' % (name, mean_score, std_score))
# Define a function for model evaluation using a test set
def evaluate_model_test_set(name, model, y_test, y_predicted):
    from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
    print('>%s - test - MAE: %.3f' % (name, mean_absolute_error(y_test, y_predicted)))
    print('>%s - test - MSE: %.3f' % (name, mean_squared_error(y_test, y_predicted)))
    print('>%s - test - R^2: %.3f' % (name, r2_score(y_test, y_predicted)))
# Evaluate the baseline model
evaluate_model_cross_validation("RF_baseline", regressor, X_train, y_train, 10)
evaluate_model_test_set("RF_baseline", regressor, y_test, y_pred)<jupyter_output>>RF_baseline - test - MAE: 5.207
>RF_baseline - test - MSE: 64.513
>RF_baseline - test - R^2: 0.621<jupyter_text>Random Search Cross ValidationWe will first perform random search training to try out a wide range of values and see what works!<jupyter_code># Define the random grid

# Number of trees in random forest
n_estimators = [int(x) for x in np.linspace(start = 200, stop = 2000, num = 10)]
# Number of features to consider at every split
max_features = ['auto', 'sqrt']
# Maximum number of levels in tree
max_depth = [int(x) for x in np.linspace(10, 110, num = 11)]
max_depth.append(None)
# Minimum number of samples required to split a node
min_samples_split = [2, 5, 10]
# Minimum number of samples required at each leaf node
min_samples_leaf = [1, 2, 4]
# Method of selecting samples for training each tree
bootstrap = [True, False]
# Create the random grid
random_grid = {'n_estimators': n_estimators,
               'max_features': max_features,
               'max_depth': max_depth,
               'min_samples_split': min_samples_split,
               'min_samples_leaf': min_samples_leaf,
               'bootstrap': bootstrap}
pprint(random_grid)<jupyter_output>{'bootstrap': [True, False],
 'max_depth': [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, None],
 'max_features': ['auto', 'sqrt'],
 'min_samples_leaf': [1, 2, 4],
 'min_samples_split': [2, 5, 10],
 'n_estimators': [200, 400, 600, 800, 1000, 1200, 1400, 1600, 1800, 2000]}<jupyter_text>As explained in the article cited above: "The most important arguments in RandomizedSearchCV are n_iter, which controls the number of different combinations to try, and cv which is the number of folds to use for cross validation (we use 100 and 3 respectively). More iterations will cover a wider search space and more cv folds reduces the chances of overfitting, but raising each will increase the run time. Machine learning is a field of trade-offs, and performance vs time is one of the most fundamental."<jupyter_code># First create the base model to tune 
regressor = RandomForestRegressor()
# Random search of parameters, using 3 fold cross validation, 
# search across 100 different combinations, and use all available cores
from sklearn.model_selection import RandomizedSearchCV
rf_random = RandomizedSearchCV(estimator = regressor,
                               param_distributions = random_grid,
                               n_iter = 100, cv = 3, verbose=2, random_state=42,
                               n_jobs = -1)
# Fit the random search model (This might take a while)
rf_random.fit(X_train, y_train)
# print the best parameters
rf_random.best_params_
# Fitting Random forest Model to the dataset using the best parameters after random search
rf_r = RandomForestRegressor(bootstrap = True,
                                  max_depth = 110,
                                  max_features ='sqrt',
                                  min_samples_leaf = 1,
                                  min_samples_split = 2,
                                  n_estimators = 1000)

rf_r.fit(X_train, y_train)

# Predicting the Test set results
y_pred = rf_r.predict(X_test)
# evaluate model after random grid search
evaluate_model_cross_validation("RF_random", rf_r, X_train, y_train, 10)
evaluate_model_test_set("RF_random", rf_r, y_test, y_pred)<jupyter_output>>RF_random - test - MAE: 4.563
>RF_random - test - MSE: 51.341
>RF_random - test - R^2: 0.699<jupyter_text>Our metrics improved from our baseline model. Let's check if we can further improve them by fine tuning our model using a systematic grid search. Grid Search with Cross Validation<jupyter_code># Applying Grid Search to find the best model and the best parameters
from sklearn.model_selection import GridSearchCV

# Number of trees in random forest
n_estimators = [800, 900, 1000, 1100, 1200]
# Number of features to consider at every split
max_features = ['sqrt']
# Maximum number of levels in tree
max_depth = [100, 110, 120]
# Minimum number of samples required to split a node
min_samples_split = [2, 3]
# Minimum number of samples required at each leaf node
min_samples_leaf = [1, 2]
# Method of selecting samples for training each tree
bootstrap = [True]
# Create the random grid
parameters = {'n_estimators': n_estimators,
               'max_features': max_features,
               'max_depth': max_depth,
               'min_samples_split': min_samples_split,
               'min_samples_leaf': min_samples_leaf,
               'bootstrap': bootstrap}
# First create the base model to tune 
regressor = RandomForestRegressor()
grid_search = GridSearchCV(estimator = regressor,
                           param_grid = parameters,
                           cv = 3, n_jobs = -1, verbose = 2)<jupyter_output><empty_output><jupyter_text>We have 5 x 1 x 3 x 2 x 2 x 2 x 1 = 60 features and 3 folds, which makes 180 combinations of settings.<jupyter_code># Fit the grid search model (This might take a while)
grid_search = grid_search.fit(X_train, y_train)
# print the best parameters
grid_search.best_params_
# Fitting Random forest Model to the dataset using the best parameters
rf_g = RandomForestRegressor(bootstrap = True,
                                  max_depth = 120,
                                  max_features ='sqrt',
                                  min_samples_leaf = 1,
                                  min_samples_split = 3,
                                  n_estimators = 900)

rf_g.fit(X_train, y_train)


# Predicting the Test set results
y_pred = rf_g.predict(X_test)
# evaluate model after grid search
evaluate_model_cross_validation("RF_grid_search", rf_g, X_train, y_train, 10)
evaluate_model_test_set("RF_grid_search", rf_g, y_test, y_pred)<jupyter_output>>RF_grid_search - test - MAE: 4.587
>RF_grid_search - test - MSE: 51.736
>RF_grid_search - test - R^2: 0.696
{'fraction_non_alphanumeric': 0.086571911340821, 'fraction_numerical': 0.026239936388033, 'mean_word_length': 3.7959961868446137, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 35, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '293085', 'license': 'MIT', 'n_tokens_mistral': 3328, 'n_tokens_neox': 3021, 'n_words': 1050, 'path': '4_Model_fine_tuning.ipynb', 'repo_name': 'linetonthat/Real_estate_Taipei_regression', 'size': '10061'}