<jupyter_start><jupyter_text>Tuning Random Forest Models Tidy Tuesday - IKEA prices On a weekly basis, a dataset is released for the R community to create visualizations and perform other analysis.  <NAME>, of R-Studio, produces a periodic tutorial on the Tidymodels framework utilizing these datasets.  Here we will replicate her analysis in Python.This week we have a dataset from IKEA.  We will attempt to predict prices using a Random Forest model.  There is a relatively small dataset 3,694 items with 13 features. Of these 13 features many are just descriptive of the item (an index, item_id, name, web link, online availability, color_availablity, designer).  We have identified 4 variables which may offer some predictive capacity, 3 numeric (depth, height, weight) and 1 categorical (category).  We will do some feature engineering with these variables, including imputation using K-Nearest-Neighbors (KNN), and One-Hot encoding.Let's begin.  Setup and Data Import<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

from sklearn.model_selection import train_test_split
from sklearn.impute import KNNImputer
from sklearn.preprocessing import OneHotEncoder

from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error

from sklearn.model_selection import RandomizedSearchCV
from sklearn.model_selection import GridSearchCV

%matplotlib inline
print("Setup Complete")<jupyter_output>Setup Complete<jupyter_text>We will import the data from the website provided and then perform for data cleaning and exploratory data anaylsis.* It is usually a good idea to inspeact the head and tail of the dataset to for a first check that the import was successful.* We will drop the `Unnamed: 0` column, as it appears to just be an index.One thing that you will notice throughout is the constant use a `.shape`.  This is used as a quick check to ensure that we are at least returning a shape that's expected.<jupyter_code>ikea_raw = pd.read_csv("https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2020/2020-11-03/ikea.csv")

ikea_raw.head()
ikea_raw.tail()
# drop this column which seems like an index
ikea = ikea_raw.drop("Unnamed: 0", axis=1)
ikea.shape<jupyter_output><empty_output><jupyter_text>After this initial cursory look at the data.  We next look to identify the completeness of our data.  We see that we have quite of few NAs, from both a table perspective and using a heatmap to see the distribution of the NAs.  Visually, it looks like we may be able to preserve some of the incomplete observations by imputing the missing values. This is beacuse the missing data is all in the dimensions of the product.  If an item is only missing a single dimension, we can attempt to impute the missing value given the other 2 dimenstions using KNN, this will be discusseed in more depth later on.  Unfortunately, we will have to drop those items which are missing 2 or more dimensions.<jupyter_code># check NAs per column
ikea.isna().sum()
# visual represenation of the NAs in the dataset
plt.figure(figsize=(10, 10))
sns.heatmap(ikea.isna());
# quickly check how many items we may be able to preseve if we attempt to impute a single missing dimension and drop those with more than two missing.
drop_index = ikea.isna().sum(axis=1) > 1
ikea.drop(ikea[drop_index].index).isna().sum()
# drop rows with 2 or more missing values
ikea.drop(ikea[drop_index].index, inplace=True)<jupyter_output><empty_output><jupyter_text>Data VisualizationsNext, we will quickly tale a look at our features as compared to our target variable.  When we look at the `price` variable, we see that it ranges from a minimum of 5 to a maximum of 9585.  To contend with this large range, we will do a log-transformation on the `price`.  We will quickly chart the target vs each or the dimensions and against the various categories.For the `categpry` feature, we see that we have 17 different categories, likely too many for our model given the small number of observations, we will take the 11 largest categories and combine the remaining categories into one called `other`.  From all these visuals, there does not appear to be great predictive value in any one feature, so we must temper our expectations of our model predictions.<jupyter_code># quick summary statistics of our target variable
ikea['price'].describe()
# log-transformation of our target variable
ikea["price_log"] = np.log(ikea["price"])
# visual the relationship between our numeric predictors vs our target
fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize = (15, 7))
ax1.scatter("depth", "price_log", data=ikea, color = "red")
ax1.set_xlabel("Depth")

ax2.scatter("height", "price_log", data=ikea, color = "green")
ax2.set_xlabel("Height")

ax3.scatter("width", "price_log", data=ikea, color = "blue")
ax3.set_xlabel("Width");
# observations in each of our categorical variable
ikea["category"].value_counts()
# reduce the categories to only 10, with at least 111 items in each category
top_11_cats = ikea["category"].value_counts().head(11).index
ikea["category"] = ikea["category"].apply(lambda cat: "Other" if cat not in top_11_cats else cat)
ikea["category"].value_counts()
# visualize how our target variable varies among the categories
plt.figure(figsize=(15, 7))
plt.xticks(rotation=90)
plt.bar("category", "price_log", data=ikea);<jupyter_output><empty_output><jupyter_text>Data PreparationWe have done some feature engineering above.  But we will need to category to ensure that our data is prepared to be entered into a model.  First, we will subset our dataset into the X (predictors or features) and y (target).  We will then create a training set, in which we will use to fit the model, and a test set, over which we will evaluate the model.**Numerical variables**  For the numerical variables, we still need to replace our missing values.  As mentioned we will use the KNN model to impute the missing values from the known dimensions.  In the KNN algorithm, we will use 3 nearest neighbors, meaning that we will find the 3 closest obeservations to determine the missing value.  The closest observations will be found by calculating the eclidian distance between each observation and then simply using the 3 closest to imput the missing value.*Note: When performing the transformation, we fit and transform the training set, but then simply use that fit to only transform the test set.***Categorical variables**For the categorical variables, we will use One-Hot Encoding to convert the categories into numeric values.  In One-Hot Encoding, we simply create a dummy feature for each category with a value of 1 or 0 for each variable depending if the category pertains to that observation or not.  Lastly, we concatenate the numeric and categorical feature back into a single DataFrame ready for modeling.<jupyter_code># create our X and Y variables
X = ikea[['category', 'depth', 'height', 'width']]
y = ikea['price_log']

X.shape, y.shape
# create our train and set datasets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

X_train.shape, X_test.shape, y_train.shape, y_test.shape
# subset our numeric features for both test and train
num_cats = ['depth', 'height', 'width']
X_train_num = X_train[num_cats]
X_test_num = X_test[num_cats]

# perform the imputation using KNN, with 3 neighbors (copy=False allows for in-place transformations)
imputer = KNNImputer(n_neighbors=3, copy=False)
imputer.fit_transform(X_train_num)
imputer.transform(X_test_num)               

X_train_num.shape, X_test_num.shape
# confirm the NAs have been replaced
X_train_num.isnull().sum(), X_test_num.isnull().sum()
# perform the One-Hot encoding for the categorical feature
One_Hot = OneHotEncoder(handle_unknown='ignore', sparse=False)

X_train_cat = X_train[['category']]
X_test_cat = X_test[['category']]

X_train_cat = pd.DataFrame(One_Hot.fit_transform(X_train_cat))
X_test_cat = pd.DataFrame(One_Hot.transform(X_test_cat))

# One-Hot Encoding drops the index, so we need to replace it
X_train_cat.index = X_train.index
X_test_cat.index = X_test.index

X_train_cat.shape, X_test_cat.shape
# recombine the transformed datasets for modeling
X_train_trans = pd.concat([X_train_num, X_train_cat], axis=1)
X_test_trans = pd.concat([X_test_num, X_test_cat], axis=1)

X_train_trans.shape, X_test_trans.shape<jupyter_output><empty_output><jupyter_text>Train the ModelNow, that our data has been transformed for modeling, we can run our initial base model.  Here we will use n_estimators of 200 (we will discuss this in the tuning section).  We will then use that fitted model to make predictions on our test data.  Those predictions will allow us to evaluate our model.  We will use a simple Mean Absolute Error for evaluation.  We must remember to transform our predictions back so that the MAE makes more intuitive sense.We didn't have high expectations for the model, and a MAE of 396 or 35% isn't very impressive.  Next, we will see if we tuning the model improves our predictive capacity.<jupyter_code># Instatiate the model, fit the model to the training data, predict on the test, and evaluate the results
rf_model = RandomForestRegressor(n_estimators=200, random_state=42)

rf_model.fit(X_train_trans, y_train)
preds = rf_model.predict(X_test_trans)

mean_absolute_error(y_test, preds)
# Remember to transform the target back for easier understanding
mean_absolute_error(np.exp(y_test), np.exp(preds)), ikea['price'].mean()<jupyter_output><empty_output><jupyter_text>Tune the ModelHere we will look to tune the model's hyperparamters.  The Random Forest model has a variety of tuning parameters, we will focus on 4 here.* n_estimators: the number of trees that will be constructed  * max_depth: maximum number of levels of each tree  * min_samples_split: minimum number of items requires to split at a node* min_samples_leaf: minimum number of items at each leaf nodeOther parameters that can be tuned:  * max_features: which is the number of features to be considered at each split* bootstrap: whether to use bootstrapping to selct the samples.When tuning you provide a range of inputs that you want the model to calculate over, based on what is chosen below this would leave $6*10*3*3$ or 540 iterations of the model.  Rather than doing each iteration which would be done using `GridSearchCV`, we will utilize `RandomizedSearchCV` which will perform a choose random searches up to the defined number of iterations, we will run 50. In addition, in order to find the best parameters, we will utilize cross-validation with 5 folds.  This will divide our training set into 5 groups, in the first run, it will train on groups (1,2,3,4) and evaluate on 5, in the 2nd run, it will train on groups (1,2,3,5) and test on 4...etc.  We will perform this cross-validation on each of the 50 randomly choosen parameter sets and the model will output the best set of parameters for us to use.<jupyter_code># define the parameters and ranges to tune over)
n_estimators = [int(x) for x in np.linspace(start=200,stop=1200, num=6)]
max_depth = [int(x) for x in np.linspace(10, 50, 10)]
min_samples_split = [2, 5, 10]
min_samples_leaf = [1, 2, 4]

random_grid = {'n_estimators': n_estimators,
              'max_depth': max_depth,
              'min_samples_split': min_samples_split,
              'min_samples_leaf': min_samples_leaf
              }
random_grid, (6*10*3*3)
# tune and fit the model to the training data
rf = RandomForestRegressor(verbose=0)
rf_random = RandomizedSearchCV(estimator=rf,
                              param_distributions=random_grid,
                              n_iter=50,
                              cv=5,
                              verbose=0,
                              random_state=42)

rf_random.fit(X_train_trans, y_train)
rf_model.estimators_
# retrieve the best parameters
rf_random.best_params_
# predict on the test set and evaluate the best model parameters
rf_best= rf_random.best_estimator_

preds = rf_best.predict(X_test_trans)

mean_absolute_error(np.exp(y_test), np.exp(preds))
plt.figure(figsize=(12, 6))
plt.scatter(y_test, preds, alpha=0.6)
plt.title("Predictions vs Target Variables")
plt.xlabel("Target")
plt.ylabel("Predictions");<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07062169849654612, 'fraction_numerical': 0.013409183258837871, 'mean_word_length': 4.899328859060403, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 30, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '316235', 'license': 'Apache-2.0', 'n_tokens_mistral': 3675, 'n_tokens_neox': 3490, 'n_words': 1559, 'path': '_notebooks/2020-12-05-Tuning-Random-Forests.ipynb', 'repo_name': 'ScottKnappNJ/FastPagesBlog', 'size': '12310'}