<jupyter_start><jupyter_text>Model engineering 003: Random Forest RegressionIn this part of the project to predict the photovoltaic production of solar cells on a roof we are considering a RandomForest regression model. We will treat this as a regression problem, not taking the temporal aspect, i.e. time series forcasting, into account. Data mining and missing value treatment of weather data from the DarkSky API and data from the photovoltaic system were covered in other notebooks:- Data mining and EDA of weather data: https://kyso.io/heiko/predicting-solar-panel-output-eda-of-photovoltaic-data- Data mining and EDA of photovoltaic data: https://kyso.io/heiko/predicting-solar-panel-output-eda-of-weather-data- Missing value treatment: https://kyso.io/heiko/predicting-solar-panel-output-missing-value-treatment-of-weather-data  MethodologyWe will apply random forest regression by following these steps:1. Load the data into one dataframe2. Select the features we will use for the prediction. We can look at the correlation matrix and remove redundant features that are correlated. Multicollinearity undermines the statistical significance of an independent variable. While it should not have a major impact on the model’s accuracy, it does affect the variance associated with the prediction, as well as, reducing the quality of the interpretation of the independent variables. In other words, the effect your data has on the model isn’t trustworthy. Your explanation of how the model takes the inputs to produce the output will not be reliable. (You can read more about this here: https://towardsdatascience.com/multicollinearity-why-is-it-a-problem-398b010b77ac). We will just drop (one of the) columns that are correlated. We could feature engineer another features that combines the correlated features, but at this point this will not be considered.3. Consider missing values. This is still part of the feature selection process. We remove features that have lots of missing values that we could not interpolate.4. Model selection. We will use RandomizedSearch and GridSearch approach to tune hyperparameters of the RandomForest model.<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import RandomizedSearchCV

from sklearn.model_selection import train_test_split
from sklearn.compose import ColumnTransformer
from sklearn.metrics import r2_score, mean_squared_error
from sklearn.model_selection import GridSearchCV

import eli5
sns.set()
%matplotlib inline
%config InlineBackend.figure_format = 'svg'<jupyter_output><empty_output><jupyter_text>Load datasetThis is the cleaned dataset from the MultiLinear model.<jupyter_code>data = pd.read_csv('../02.Prepared_data/dataset.Model_eng.001.csv')<jupyter_output><empty_output><jupyter_text>Feature selectionSelect the features in a pipeline as for the linear model.<jupyter_code># X = df_cleaned.iloc[:, 1:].values
X = data.iloc[:, 1:]
X = X.reset_index(drop=True)
# y = df_cleaned.iloc[:, 0].values
y = data.iloc[:, 0]
y = y.reset_index(drop=True)<jupyter_output><empty_output><jupyter_text>Numeric features: StandardScalerStandardize features by removing the mean and scaling to unit variance<jupyter_code>numeric_features = ['apparentTemperatureHigh', 'precipProbability', 'uvIndex', 'precipIntensityMax_cm', 'sun_uptime']
numeric_transformer = Pipeline(steps=[
    ('scaler', StandardScaler())])<jupyter_output><empty_output><jupyter_text>Categorical features: OneHotEncoderThis preprocessing step will encode the `season` and `precipType` column values into a vector of length 3. For example, winter will be encoded as [1,0,0], summer as [0,1,0], and so on for the other seasons.<jupyter_code>categorical_features = ['season', 'precipType']
categorical_transformer = Pipeline(steps=[
    ('onehot', OneHotEncoder())])

preprocessor = ColumnTransformer(
    transformers=[
        ('num', numeric_transformer, numeric_features),
        ('cat', categorical_transformer, categorical_features)])
clf = Pipeline(steps=[('preprocessor', preprocessor),
                      ('classifier', RandomForestRegressor())])

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)<jupyter_output><empty_output><jupyter_text>Randomized and GridSearch to tune the hyperparametersThe idea behind a RandomForest is to get N estimators from one estimator (one tree). Random forest consist of a number of estimators, which are decision trees. In the sampling process that is called bagging, the dataset is repeatedly sampled (with or without replacement). Then in a parallel operation, a decision tree is fitted on each of the individual datasets. In the final step, the average of all the individual trees outcome is taken as the regression value. This tackles the problem of overfitting of a single tree, that can happen with a single DecisionTree alone. RandomForests show good performance on many problems, including nonlinear ones. However, they offer limited interpretability and overfitting can be a problem. The number of trees must be chosen carefully.To tune the hyperparameters, first there will be a randomized search to find where the improved parameters may lie. That limits the feature space for the grid search. In a second step, a grid search around these parameters is performed.Grid Search is as an exhaustive search for selecting a model. Every combination of hyperparameter values is tried which can be very inefficient. By contrast, Random Search sets up a grid of hyperparameter values and selects random combinations to train the model and score. This allows control of the number of parameter combinations that are attempted.<jupyter_code># first access the parameter keys of the individual estimators
clf.get_params().keys()
# Number of trees in random forest
n_estimators = [int(x) for x in np.linspace(start = 100, stop = 2000, num = 10)]
# Number of features to consider at every split
max_features = ['auto', 'sqrt']
# Maximum number of levels in tree
max_depth = [int(x) for x in np.linspace(10, 150, num = 11)]
max_depth.append(None)
# Minimum number of samples required to split a node
min_samples_split = [2, 5, 10]
# Minimum number of samples required at each leaf node
min_samples_leaf = [1, 2, 4]
# Method of selecting samples for training each tree
bootstrap = [True, False]

# Create the random grid
random_grid = {'classifier__n_estimators': n_estimators,
               'classifier__max_features': max_features,
               'classifier__max_depth': max_depth,
               'classifier__min_samples_split': min_samples_split,
               'classifier__min_samples_leaf': min_samples_leaf,
               'classifier__bootstrap': bootstrap}


# Use the random grid to search for best hyperparameters
# First create the base model to tune
# clf = RandomForestClassifier()
# Random search of parameters, using 3 fold cross validation, 
# search across 100 different combinations, and use all available cores
clf_random = RandomizedSearchCV(estimator=clf, param_distributions=random_grid, n_iter=100, cv=3, verbose=2, random_state=42, n_jobs=-1)
# Fit the random search model
clf_random.fit(X_train, y_train)
clf_random.best_params_
from sklearn.model_selection import GridSearchCV


# Number of trees in random forest
n_estimators = [1500, 2000, 2500]
# Number of features to consider at every split
max_features = ['sqrt']
# Maximum number of levels in tree
max_depth = [5, 10, 15]
# Minimum number of samples required to split a node
min_samples_split = [4,5,6]
# Minimum number of samples required at each leaf node
min_samples_leaf = [1]
# Method of selecting samples for training each tree
bootstrap = [True]

# Create the random grid
param_grid = {'classifier__n_estimators': n_estimators,
               'classifier__max_features': max_features,
               'classifier__max_depth': max_depth,
               'classifier__min_samples_split': min_samples_split,
               'classifier__min_samples_leaf': min_samples_leaf,
               'classifier__bootstrap': bootstrap}

# Instantiate the grid search model
grid_search = GridSearchCV(estimator=clf, param_grid=param_grid, 
                          cv=3, n_jobs=-1, verbose=2)
grid_search.fit(X_train, y_train)

best_model = grid_search.best_estimator_
best_model.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Evaluate the model: r2 score and MSE<jupyter_code>y_pred = best_model.predict(X_test)
r2_score(y_test, y_pred)
mean_squared_error(y_test, y_pred)
importance = best_model.named_steps['classifier'].feature_importances_
# summarize feature importance
for i,v in enumerate(importance):
    print('Feature: %0d, Score: %.5f' % (i,v))
    
# get feature names
onehot_columns = list(best_model.named_steps['preprocessor'].named_transformers_['cat'].named_steps['onehot'].get_feature_names(input_features=categorical_features))
numeric_features_list = list(numeric_features)
numeric_features_list.extend(onehot_columns)


# plot feature importance
fig, ax = plt.subplots(figsize=(6,4))
plt.bar([x for x in range(len(importance))], importance, align='center')
plt.xticks(np.arange(0, len(importance)), numeric_features_list, rotation=90)
plt.ylabel('Feature importance from 1500 trees')
plt.tight_layout()
plt.savefig('random_forest.png', dpi=200)
plt.show()<jupyter_output>Feature: 0, Score: 0.25617
Feature: 1, Score: 0.12860
Feature: 2, Score: 0.14004
Feature: 3, Score: 0.10441
Feature: 4, Score: 0.24764
Feature: 5, Score: 0.02248
Feature: 6, Score: 0.03981
Feature: 7, Score: 0.01077
Feature: 8, Score: 0.03364
Feature: 9, Score: 0.00698
Feature: 10, Score: 0.00604
Feature: 11, Score: 0.00341
{'fraction_non_alphanumeric': 0.07675126903553299, 'fraction_numerical': 0.02, 'mean_word_length': 5.3926022063595065, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '349508', 'license': 'MIT', 'n_tokens_mistral': 2963, 'n_tokens_neox': 2754, 'n_words': 1055, 'path': '2020-04-16.solarAnalytics/04.Analysis/2020-06-25.Model_eng.003.RandomForest.ipynb', 'repo_name': 'kromerh/solarAnalytics', 'size': '9850'}