<jupyter_start><jupyter_text>Assignment - Decision Trees and Random Forests![](https://i.imgur.com/3sw1fY9.jpg)In this assignment, you'll continue building on the previous assignment to predict the price of a house using information like its location, area, no. of rooms etc. You'll use the dataset from the [House Prices - Advanced Regression Techniques](https://www.kaggle.com/c/house-prices-advanced-regression-techniques) competition on [Kaggle](https://kaggle.com). We'll follow a step-by-step process:1. Download and prepare the dataset for training2. Train, evaluate and interpret a decision tree3. Train, evaluate and interpret a random forest4. Tune hyperparameters to improve the model5. Make predictions and save the modelAs you go through this notebook, you will find a **???** in certain places. Your job is to replace the **???** with appropriate code or values, to ensure that the notebook runs properly end-to-end and your machine learning model is trained properly without errors. **Guidelines**1. Make sure to run all the code cells in order. Otherwise, you may get errors like `NameError` for undefined variables.2. Do not change variable names, delete cells, or disturb other existing code. It may cause problems during evaluation.3. In some cases, you may need to add some code cells or new statements before or after the line of code containing the **???**. 4. Since you'll be using a temporary online service for code execution, save your work by running `jovian.commit` at regular intervals.5. Review the "Evaluation Criteria" for the assignment carefully and make sure your submission meets all the criteria.6. Questions marked **(Optional)** will not be considered for evaluation and can be skipped. They are for your learning.7. It's okay to ask for help & discuss ideas on the [community forum](https://jovian.ai/forum/c/zero-to-gbms/gbms-assignment-2/99), but please don't post full working code, to give everyone an opportunity to solve the assignment on their own.**Important Links**:- Make a submission here: https://jovian.ai/learn/machine-learning-with-python-zero-to-gbms/assignment/assignment-2-decision-trees-and-random-forests- Ask questions, discuss ideas and get help here: https://jovian.ai/forum/c/zero-to-gbms/gbms-assignment-2/99- Review this Jupyter notebook: https://jovian.ai/aakashns/sklearn-decision-trees-random-forests  How to Run the Code and Save Your Work**Option 1: Running using free online resources (1-click, recommended):** The easiest way to start executing the code is to click the **Run** button at the top of this page and select **Run on Binder**. This will set up a cloud-based Jupyter notebook server and allow you to modify/execute the code.**Option 2: Running on your computer locally:** To run the code on your computer locally, you'll need to set up [Python](https://www.python.org), download the notebook and install the required libraries. Click the **Run** button at the top of this page, select the **Run Locally** option, and follow the instructions.**Saving your work**: You can save a snapshot of the assignment to your [Jovian](https://jovian.ai) profile, so that you can access it later and continue your work. Keep saving your work by running `jovian.commit` from time to time.<jupyter_code>!pip install jovian --upgrade --quiet
import jovian
jovian.commit(project='python-random-forests-assignment', privacy='secret')<jupyter_output><empty_output><jupyter_text>Let's begin by installing the required libraries.<jupyter_code>!pip install opendatasets scikit-learn plotly folium --upgrade --quiet
!pip install pandas numpy matplotlib seaborn --quiet<jupyter_output><empty_output><jupyter_text>Download and prepare the dataset for training<jupyter_code>import os
from zipfile import ZipFile
from urllib.request import urlretrieve

dataset_url = 'https://github.com/JovianML/opendatasets/raw/master/data/house-prices-advanced-regression-techniques.zip'
urlretrieve(dataset_url, 'house-prices.zip')
with ZipFile('house-prices.zip') as f:
    f.extractall(path='house-prices')
    
os.listdir('house-prices')
import pandas as pd
pd.options.display.max_columns = 200
pd.options.display.max_rows = 200

prices_df = pd.read_csv('house-prices/train.csv')
prices_df
import numpy as np
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import MinMaxScaler, OneHotEncoder
from sklearn.model_selection import train_test_split

# Identify input and target columns
input_cols, target_col = prices_df.columns[1:-1], prices_df.columns[-1]
inputs_df, targets = prices_df[input_cols].copy(), prices_df[target_col].copy()

# Identify numeric and categorical columns
numeric_cols = prices_df[input_cols].select_dtypes(include=np.number).columns.tolist()
categorical_cols = prices_df[input_cols].select_dtypes(include='object').columns.tolist()

# Impute and scale numeric columns
imputer = SimpleImputer().fit(inputs_df[numeric_cols])
inputs_df[numeric_cols] = imputer.transform(inputs_df[numeric_cols])
scaler = MinMaxScaler().fit(inputs_df[numeric_cols])
inputs_df[numeric_cols] = scaler.transform(inputs_df[numeric_cols])

# One-hot encode categorical columns
encoder = OneHotEncoder(sparse=False, handle_unknown='ignore').fit(inputs_df[categorical_cols])
encoded_cols = list(encoder.get_feature_names(categorical_cols))
inputs_df[encoded_cols] = encoder.transform(inputs_df[categorical_cols])

# Create training and validation sets
train_inputs, val_inputs, train_targets, val_targets = train_test_split(
    inputs_df[numeric_cols + encoded_cols], targets, test_size=0.25, random_state=42)<jupyter_output><empty_output><jupyter_text>Let's save our work before continuing.<jupyter_code>jovian.commit()<jupyter_output><empty_output><jupyter_text>Decision Tree > **QUESTION 1**: Train a decision tree regressor using the training set.<jupyter_code>from sklearn.tree import DecisionTreeRegressor
# Create the model
tree = DecisionTreeRegressor(random_state=42)
# Fit the model to the training data
%time
tree.fit(train_inputs, train_targets)<jupyter_output>CPU times: user 4 µs, sys: 1e+03 ns, total: 5 µs
Wall time: 6.91 µs<jupyter_text>Let's save our work before continuing.<jupyter_code>jovian.commit()<jupyter_output><empty_output><jupyter_text>> **QUESTION 2**: Generate predictions on the training and validation sets using the trained decision tree, and compute the RMSE loss.<jupyter_code>from sklearn.metrics import mean_squared_error
tree_train_preds = tree.predict(train_inputs)
tree_train_rmse = mean_squared_error(train_targets, tree_train_preds)
tree_val_preds = tree.predict(val_inputs)
tree_val_rmse = mean_squared_error(val_targets, tree_val_preds)
print('Train RMSE: {}, Validation RMSE: {}'.format(tree_train_rmse, tree_val_rmse))<jupyter_output>Train RMSE: 0.0, Validation RMSE: 1429057134.4054794<jupyter_text>Let's save our work before continuing.<jupyter_code>jovian.commit()<jupyter_output><empty_output><jupyter_text>> **QUESTION 3**: Visualize the decision tree (graphically and textually) and display feature importances as a graph. Limit the maximum depth of graphical visualization to 3 levels.<jupyter_code>import matplotlib.pyplot as plt
from sklearn.tree import plot_tree, export_text
import seaborn as sns
sns.set_style('darkgrid')
%matplotlib inline
plt.figure(figsize=(30,15))

# Visualize the tree graphically using plot_tree
importance_df = pd.DataFrame({
    "feature": train_inputs.columns,
    "importance": tree.feature_importances_
}).sort_values("importance", ascending=False)
plt.title("Feature Importance")
sns.barplot(data=importance_df.head(10), x="importance", y="feature");
# Visualize the tree textually using export_text
tree_text = export_text(tree, max_depth=10, feature_names=list(train_inputs.columns))
# Display the first few lines
print(tree_text[:2000])
# Check feature importance
tree_importances = tree.feature_importances_
tree_importance_df = pd.DataFrame({
    'feature': train_inputs.columns,
    'importance': tree_importances
}).sort_values('importance', ascending=False)
tree_importance_df
plt.title('Decision Tree Feature Importance')
sns.barplot(data=tree_importance_df.head(10), x='importance', y='feature');<jupyter_output><empty_output><jupyter_text>Let's save our work before continuing.<jupyter_code>jovian.commit()<jupyter_output><empty_output><jupyter_text>Random Forests > **QUESTION 4**: Train a random forest regressor using the training set.<jupyter_code>from sklearn.ensemble import RandomForestRegressor
# Create the model
rf1 = RandomForestRegressor(n_jobs=-1, random_state=42)
# Fit the model
%time
rf1.fit(train_inputs, train_targets)<jupyter_output>CPU times: user 11 µs, sys: 2 µs, total: 13 µs
Wall time: 23.4 µs<jupyter_text>Let's save our work before continuing.<jupyter_code>jovian.commit()<jupyter_output><empty_output><jupyter_text>> **QUESTION 5**: Make predictions using the random forest regressor.<jupyter_code>rf1_train_preds = rf1.predict(train_inputs)
rf1_train_rmse = mean_squared_error(train_targets, rf1_train_preds)
rf1_val_preds = rf1.predict(val_inputs)
rf1_val_rmse = mean_squared_error(val_targets, rf1_val_preds)
print('Train RMSE: {}, Validation RMSE: {}'.format(rf1_train_rmse, rf1_val_rmse))<jupyter_output>Train RMSE: 140762048.72081444, Validation RMSE: 774511050.5034367<jupyter_text>Let's save our work before continuing.<jupyter_code>jovian.commit()<jupyter_output><empty_output><jupyter_text>Hyperparameter TuningLet us now tune the hyperparameters of our model. You can find the hyperparameters for `RandomForestRegressor` here: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.htmlHyperparameters are use Let's define a helper function `test_params` which can test the given value of one or more hyperparameters.<jupyter_code>def test_params(**params):
    model = RandomForestRegressor(random_state=42, n_jobs=-1, **params).fit(train_inputs, train_targets)
    train_rmse = mean_squared_error(model.predict(train_inputs), train_targets, squared=False)
    val_rmse = mean_squared_error(model.predict(val_inputs), val_targets, squared=False)
    return train_rmse, val_rmse<jupyter_output><empty_output><jupyter_text>It can be used as follows:<jupyter_code>test_params(n_estimators=20, max_depth=20)
test_params(n_estimators=50, max_depth=10, min_samples_leaf=4, max_features=0.4)<jupyter_output><empty_output><jupyter_text>Let's also define a helper function to test and plot different values of a single parameter.<jupyter_code>def test_param_and_plot(param_name, param_values):
    train_errors, val_errors = [], [] 
    for value in param_values:
        params = {param_name: value}
        train_rmse, val_rmse = test_params(**params)
        train_errors.append(train_rmse)
        val_errors.append(val_rmse)
    plt.figure(figsize=(10,6))
    plt.title('Overfitting curve: ' + param_name)
    plt.plot(param_values, train_errors, 'b-o')
    plt.plot(param_values, val_errors, 'r-o')
    plt.xlabel(param_name)
    plt.ylabel('RMSE')
    plt.legend(['Training', 'Validation'])
test_param_and_plot('max_depth', [5, 10, 15, 20, 25, 30, 35, 40])<jupyter_output><empty_output><jupyter_text>From the above graph, it appears that the best value for `max_depth` is around 20, beyond which the model starts to overfit. Let's save our work before continuing.<jupyter_code>jovian.commit()<jupyter_output><empty_output><jupyter_text>> **QUESTION 6**: Use the `test_params` and `test_param_and_plot` functions to experiment with different values of the  hyperparmeters like `n_estimators`, `max_depth`, `min_samples_split`, `min_samples_leaf`, `min_weight_fraction_leaf`, `max_features`, `max_leaf_nodes`, `min_impurity_decrease`, `min_impurity_split` etc. You can learn more about the hyperparameters here: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html<jupyter_code>test_params(max_depth=20)
test_params(max_depth=21)
test_param_and_plot('max_leaf_nodes', [2**5, 2**10, 2**15])<jupyter_output><empty_output><jupyter_text>Let's save our work before continuing.<jupyter_code>jovian.commit()<jupyter_output><empty_output><jupyter_text>Training the Best Model> **QUESTION 7**: Train a random forest regressor model with your best hyperparameters to minimize the validation loss.<jupyter_code># Create the model with custom hyperparameters
rf2 = RandomForestRegressor(max_depth=21, n_jobs=-1, random_state=42)
# Train the model
rf2.fit(train_inputs, train_targets)<jupyter_output><empty_output><jupyter_text>Let's save our work before continuing.<jupyter_code>jovian.commit()<jupyter_output><empty_output><jupyter_text>> **QUESTION 8**: Make predictions and evaluate your final model. If you're unhappy with the results, modify the hyperparameters above and try again.<jupyter_code>rf2_train_preds = rf2.predict(train_inputs)
rf2_train_rmse = mean_squared_error(train_targets, rf2_train_preds)
rf2_val_preds = rf2.predict(val_inputs)
rf2_val_rmse = mean_squared_error(val_targets, rf2_val_preds)
print('Train RMSE: {}, Validation RMSE: {}'.format(rf2_train_rmse, rf2_val_rmse))<jupyter_output>Train RMSE: 140939253.27866283, Validation RMSE: 765659190.1739577<jupyter_text>Let's also view and plot the feature importances.<jupyter_code>rf2_importance_df = pd.DataFrame({
    'feature': train_inputs.columns,
    'importance': rf2.feature_importances_
}).sort_values('importance', ascending=False)
sns.barplot(data=rf2_importance_df.head(), x='importance', y='feature')
rf2_importance_df<jupyter_output><empty_output><jupyter_text>Let's save our work before continuing.<jupyter_code>jovian.commit()<jupyter_output><empty_output><jupyter_text>Make a SubmissionTo make a submission, just execute the following cell:<jupyter_code>jovian.submit('zerotogbms-a2')<jupyter_output><empty_output><jupyter_text>You can also submit your Jovian notebook link on the assignment page: https://jovian.ai/learn/machine-learning-with-python-zero-to-gbms/assignment/assignment-2-decision-trees-and-random-forestsMake sure to review the evaluation criteria carefully. You can make any number of submissions, and only your final submission will be evalauted.Ask questions, discuss ideas and get help here: https://jovian.ai/forum/c/zero-to-gbms/gbms-assignment-2/99NOTE: **The rest of this assignment is optional.**  Making Predictions on the Test SetLet's make predictions on the test set provided with the data.<jupyter_code>test_df = pd.read_csv('house-prices/test.csv')
test_df<jupyter_output><empty_output><jupyter_text>First, we need to reapply all the preprocessing steps.<jupyter_code>test_df[numeric_cols] = imputer.transform(test_df[numeric_cols])
test_df[numeric_cols] = scaler.transform(test_df[numeric_cols])
test_df[encoded_cols] = encoder.transform(test_df[categorical_cols])
test_inputs = test_df[numeric_cols + encoded_cols]<jupyter_output><empty_output><jupyter_text>We can now make predictions using our final model.<jupyter_code>test_preds = rf2.predict(test_inputs)
submission_df = pd.read_csv('house-prices/sample_submission.csv')
submission_df<jupyter_output><empty_output><jupyter_text>Let's replace the values of the `SalePrice` column with our predictions.<jupyter_code>submission_df['SalePrice'] = test_preds<jupyter_output><empty_output><jupyter_text>Let's save it as a CSV file and download it.<jupyter_code>submission_df.to_csv('submission.csv', index=False)
from IPython.display import FileLink
FileLink('submission.csv') # Doesn't work on Colab, use the file browser instead to download the file.<jupyter_output><empty_output><jupyter_text>We can now submit this file to the competition: https://www.kaggle.com/c/house-prices-advanced-regression-techniques/submissions![](https://i.imgur.com/6h2vXRq.png) > **(OPTIONAL) QUESTION**: Submit your predictions to the competition. Experiment with different models, feature engineering strategies and hyperparameters and try to reach the top 10% on the leaderboard. Let's save our work before continuing.<jupyter_code>jovian.commit()<jupyter_output><empty_output><jupyter_text>Making Predictions on Single Inputs<jupyter_code>def predict_input(model, single_input):
    input_df = pd.DataFrame([single_input])
    input_df[numeric_cols] = imputer.transform(input_df[numeric_cols])
    input_df[numeric_cols] = scaler.transform(input_df[numeric_cols])
    input_df[encoded_cols] = encoder.transform(input_df[categorical_cols].values)
    return model.predict(input_df[numeric_cols + encoded_cols])[0]
sample_input = { 'MSSubClass': 20, 'MSZoning': 'RL', 'LotFrontage': 77.0, 'LotArea': 9320,
 'Street': 'Pave', 'Alley': None, 'LotShape': 'IR1', 'LandContour': 'Lvl', 'Utilities': 'AllPub',
 'LotConfig': 'Inside', 'LandSlope': 'Gtl', 'Neighborhood': 'NAmes', 'Condition1': 'Norm', 'Condition2': 'Norm',
 'BldgType': '1Fam', 'HouseStyle': '1Story', 'OverallQual': 4, 'OverallCond': 5, 'YearBuilt': 1959,
 'YearRemodAdd': 1959, 'RoofStyle': 'Gable', 'RoofMatl': 'CompShg', 'Exterior1st': 'Plywood',
 'Exterior2nd': 'Plywood', 'MasVnrType': 'None','MasVnrArea': 0.0,'ExterQual': 'TA','ExterCond': 'TA',
 'Foundation': 'CBlock','BsmtQual': 'TA','BsmtCond': 'TA','BsmtExposure': 'No','BsmtFinType1': 'ALQ',
 'BsmtFinSF1': 569,'BsmtFinType2': 'Unf','BsmtFinSF2': 0,'BsmtUnfSF': 381,
 'TotalBsmtSF': 950,'Heating': 'GasA','HeatingQC': 'Fa','CentralAir': 'Y','Electrical': 'SBrkr', '1stFlrSF': 1225,
 '2ndFlrSF': 0, 'LowQualFinSF': 0, 'GrLivArea': 1225, 'BsmtFullBath': 1, 'BsmtHalfBath': 0, 'FullBath': 1,
 'HalfBath': 1, 'BedroomAbvGr': 3, 'KitchenAbvGr': 1,'KitchenQual': 'TA','TotRmsAbvGrd': 6,'Functional': 'Typ',
 'Fireplaces': 0,'FireplaceQu': np.nan,'GarageType': np.nan,'GarageYrBlt': np.nan,'GarageFinish': np.nan,'GarageCars': 0,
 'GarageArea': 0,'GarageQual': np.nan,'GarageCond': np.nan,'PavedDrive': 'Y', 'WoodDeckSF': 352, 'OpenPorchSF': 0,
 'EnclosedPorch': 0,'3SsnPorch': 0, 'ScreenPorch': 0, 'PoolArea': 0, 'PoolQC': np.nan, 'Fence': np.nan, 'MiscFeature': 'Shed',
 'MiscVal': 400, 'MoSold': 1, 'YrSold': 2010, 'SaleType': 'WD', 'SaleCondition': 'Normal'}
predicted_price = predict_input(rf2, sample_input)
print('The predicted sale price of the house is ${}'.format(predicted_price))<jupyter_output>The predicted sale price of the house is $123618.0<jupyter_text>> **EXERCISE**: Change the sample input above and make predictions. Try different examples and try to figure out which columns have a big impact on the sale price. Hint: Look at the feature importance to decide which columns to try.  Saving the Model<jupyter_code>import joblib
house_prices_rf = {
    'model': rf2,
    'imputer': imputer,
    'scaler': scaler,
    'encoder': encoder,
    'input_cols': input_cols,
    'target_col': target_col,
    'numeric_cols': numeric_cols,
    'categorical_cols': categorical_cols,
    'encoded_cols': encoded_cols
}
joblib.dump(house_prices_rf, 'house_prices_rf.joblib')<jupyter_output><empty_output><jupyter_text>Let's save our work before continuing.<jupyter_code>jovian.commit(outputs=['house_prices_rf.joblib'])<jupyter_output><empty_output><jupyter_text>Predicting the Logarithm of Sale Price > **(OPTIONAL) QUESTION**: In the [original Kaggle competition](https://www.kaggle.com/c/house-prices-advanced-regression-techniques/overview/evaluation), the model is evaluated by computing the Root Mean Squared Error on the logarithm of the sale price. Try training a random forest to predict the logarithm of the sale price, instead of the actual sales price and see if the results you obtain are better than the models trained above.<jupyter_code>!pip install --upgrade pip<jupyter_output>Requirement already satisfied: pip in /home/reymond/anaconda3/lib/python3.7/site-packages (21.2.4)
{'fraction_non_alphanumeric': 0.1117844774903239, 'fraction_numerical': 0.018079038500712977, 'mean_word_length': 7.154900332225914, 'pattern_counts': {'":': 2, '<': 138, '<?xml version=': 0, '>': 149, 'https://': 17, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '36', 'dir': 'jupyter-structured-clean-dedup', 'id': '581980', 'license': 'MIT', 'n_tokens_mistral': 6777, 'n_tokens_neox': 6332, 'n_words': 1656, 'path': 'Machine_Learning_with_Python_Zero_to_GBMs/Assignment_2_Decision_Trees_and_Random_Forests/python-random-forests-assignment.ipynb', 'repo_name': 'Rey-commits/Jovian.AI', 'size': '19636'}