<jupyter_start><jupyter_text>Random Forest Model without Automatic Feature Engineering In this notebook, we will focus on achieving better performance by choosing the most important features through **Feature Reduction**  1. Loading Data and Packages<jupyter_code># Importing Libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
# Importing the dataset
column_names = ['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'MEDV']
dataset = pd.read_csv('housing.csv', delim_whitespace=True, names=column_names)
dataset.describe()<jupyter_output><empty_output><jupyter_text>2. Analyzing the data<jupyter_code># visualize the dataset
import seaborn as sns
import matplotlib.pyplot as plt

fig, axs = plt.subplots(ncols=7, nrows=2, figsize=(20, 10))
index = 0
axs = axs.flatten()

# draw boxplots with vertical orientation
for k,v in dataset.items():
    sns.boxplot(y=k, data=dataset, ax=axs[index])
    index += 1
plt.tight_layout(pad=0.4, w_pad=1, h_pad=5.0)<jupyter_output>/usr/local/lib/python3.6/dist-packages/seaborn/categorical.py:454: FutureWarning: remove_na is deprecated and is a private function. Do not use.
  box_data = remove_na(group_data)<jupyter_text>Columns like CRIM, ZN, RM, B seems to have outliers. Let's see the outliers percentage in every column.<jupyter_code># Let's see the outliers percentage in every column.
for k, v in dataset.items():
        q1 = v.quantile(0.25)
        q3 = v.quantile(0.75)
        irq = q3 - q1
        v_col = v[(v <= q1 - 1.5 * irq) | (v >= q3 + 1.5 * irq)]
        perc = np.shape(v_col)[0] * 100.0 / np.shape(dataset)[0]
        print("Column %s outliers = %.2f%%" % (k, perc))<jupyter_output>Column CRIM outliers = 13.04%
Column ZN outliers = 13.44%
Column INDUS outliers = 0.00%
Column CHAS outliers = 100.00%
Column NOX outliers = 0.00%
Column RM outliers = 5.93%
Column AGE outliers = 0.00%
Column DIS outliers = 0.99%
Column RAD outliers = 0.00%
Column TAX outliers = 0.00%
Column PTRATIO outliers = 2.96%
Column B outliers = 15.22%
Column LSTAT outliers = 1.38%
Column MEDV outliers = 7.91%<jupyter_text>3. Data Preprocessing Let's remove MEDV outliers (MEDV = 50.0)  and RM outliers (RM = 8.78) .<jupyter_code>dataset = dataset[~(dataset['MEDV'] >= 50.0)]
dataset = dataset[~(dataset['RM'] >= 8.78)]
print(np.shape(dataset))
y = dataset['MEDV']
X = dataset.drop('MEDV', axis = 1)
dataset.head()
# Splitting the dataset into the Training set and Test set
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)<jupyter_output><empty_output><jupyter_text>4. Fitting the Base Model<jupyter_code># Fitting the RandomForest Regression Model to the dataset
from sklearn.ensemble import RandomForestRegressor
regressor = RandomForestRegressor(n_estimators=1000, random_state=42)
regressor.fit(X_train, y_train)
# Predicting a new result
y_pred = regressor.predict(X_test)
# Calculating accuracy using r2_score
from sklearn.metrics import r2_score
train_score = round(regressor.score(X_train, y_train)*100,2)
test_score = round(r2_score(y_test, y_pred)*100,2)
print('----------------Model Performance---------------')
print("Train_accuracy_Base :" + str(train_score))
print("Test_accuracy_Base :" + str(test_score))<jupyter_output>----------------Model Performance---------------
Train_accuracy_Base :98.23
Test_accuracy_Base :89.46<jupyter_text>5. Feature Reduction From previous experience and the graphs produced at the beginning, we know that some features are not useful for our problem. To reduce the number of features, which will reduce runtime, hopefully without significantly reducing performance, we can examine the feature importances from the random forest. **Feature Importances**<jupyter_code># List of features for later use
feature_list = list(X.columns)

# Get numerical feature importances
importances = list(regressor.feature_importances_)

# List of tuples with variable and importance
feature_importances = [(feature, round(importance, 2)) for feature, importance in zip(feature_list, importances)]

# Sort the feature importances by most important first
feature_importances = sorted(feature_importances, key = lambda x: x[1], reverse = True)
# Print out the feature and importances 
[print('Variable: {:20} Importance: {}'.format(*pair)) for pair in feature_importances];<jupyter_output>Variable: LSTAT                Importance: 0.44
Variable: RM                   Importance: 0.39
Variable: CRIM                 Importance: 0.04
Variable: NOX                  Importance: 0.03
Variable: DIS                  Importance: 0.02
Variable: PTRATIO              Importance: 0.02
Variable: INDUS                Importance: 0.01
Variable: AGE                  Importance: 0.01
Variable: TAX                  Importance: 0.01
Variable: B                    Importance: 0.01
Variable: ZN                   Importance: 0.0
Variable: CHAS                 Importance: 0.0
Variable: RAD                  Importance: 0.0<jupyter_text>**Visualize Feature Importances**<jupyter_code># Reset style 
plt.style.use('fivethirtyeight')

# list of x locations for plotting
x_values = list(range(len(importances)))

# Make a bar chart
plt.bar(x_values, importances, orientation = 'vertical', color = 'r', edgecolor = 'k', linewidth = 1)

# Tick labels for x axis
plt.xticks(x_values, feature_list, rotation='vertical', fontsize=10)

# Axis labels and title
plt.ylabel('Importance', fontsize=12)
plt.xlabel('Variable', fontsize=12)
plt.title('Variable Importances', fontsize=16)
# List of features sorted from most to least important
sorted_importances = [importance[1] for importance in feature_importances]
sorted_features = [importance[0] for importance in feature_importances]
# Cumulative importances
cumulative_importances = np.cumsum(sorted_importances)

# Make a line graph
plt.plot(x_values, cumulative_importances, 'g-')

# Draw line at 95% of importance retained
plt.hlines(y = 0.97, xmin=0, xmax=len(sorted_importances), color = 'r', linestyles = 'dashed')

# Format x ticks and labels
plt.xticks(x_values, sorted_features, rotation = 'vertical')

# Axis labels and title
plt.xlabel('Variable')
plt.ylabel('Cumulative Importance') 
plt.title('Cumulative Importances')<jupyter_output><empty_output><jupyter_text>**Limit Number of Features** We will now reduce the number of features in use by the model to only those required to account for 97% of the importance. The same number of features must be used in the training and testing sets.<jupyter_code># Find number of features for cumulative importance of 97%
# Add 1 because Python is zero-indexed
print('Number of features for 97% importance:', np.where(cumulative_importances > 0.97)[0][0] + 1)
# Extract the names of the most important features
important_feature_names = [feature[0] for feature in feature_importances[0:9]]
# Find the columns of the most important features
important_indices = [feature_list.index(feature) for feature in important_feature_names]

# Create training and testing sets with only the important features
important_train_features = X_train.iloc[:, important_indices]
important_test_features = X_test.iloc[:, important_indices]

# Sanity check on operations
print('Important train features shape:', important_train_features.shape)
print('Important test features shape:', important_test_features.shape)<jupyter_output>Important train features shape: (391, 9)
Important test features shape: (98, 9)<jupyter_text>**Training on Important Features**<jupyter_code># Train the model on only the important features
regressor.fit(important_train_features, y_train);<jupyter_output><empty_output><jupyter_text>**Evaluate on Important features**<jupyter_code># Make predictions on test data
predictions = regressor.predict(important_test_features)
# Calculating accuracy using r2_score
from sklearn.metrics import r2_score
train_score = round(regressor.score(important_train_features, y_train)*100,2)
test_score = round(r2_score(y_test, predictions)*100,2)
print('----------------Model Performance---------------')
print("Train_accuracy_Reduced :" + str(train_score))
print("Test_accuracy_Reduced :" + str(test_score))<jupyter_output>----------------Model Performance---------------
Train_accuracy_Reduced :98.2
Test_accuracy_Reduced :89.0<jupyter_text>Using only the 9 most important features (instead of all 13) results in a minor decrease in accuracy by 0.46 percent. For some models, decreasing the number of features can increase performance and therefore should be done. However, in other situations, performance will decrease but run time will also decrease. The final decision on how many features to retain will therefore be a trade-off between accuracy and run time.  6. Compare Trade-Offs<jupyter_code># Use time library for run time evaluation
import time

# All features training and testing time
all_features_time = []

# Do 10 iterations and take average for all features
for _ in range(10):
    start_time = time.time()
    regressor.fit(X_train, y_train)
    all_features_predictions = regressor.predict(X_test)
    end_time = time.time()
    all_features_time.append(end_time - start_time)

all_features_time = np.mean(all_features_time)
print('All features total training and testing time:', round(all_features_time, 2), 'seconds.')
# Time training and testing for reduced feature set
reduced_features_time = []

# Do 10 iterations and take average
for _ in range(10):
    start_time = time.time()
    regressor.fit(important_train_features, y_train)
    reduced_features_predictions = regressor.predict(important_test_features)
    end_time = time.time()
    reduced_features_time.append(end_time - start_time)

reduced_features_time = np.mean(reduced_features_time)
print('Reduced features total training and testing time:', round(reduced_features_time, 2), 'seconds.')<jupyter_output>Reduced features total training and testing time: 2.11 seconds.<jupyter_text>**Accuracy vs Run-Time**<jupyter_code>all_accuracy =  round(r2_score(y_test, y_pred)*100,2)
reduced_accuracy = round(r2_score(y_test, predictions)*100,2)

comparison = pd.DataFrame({'features': ['all (17)', 'reduced (5)'], 
                           'run_time': [round(all_features_time, 2), round(reduced_features_time, 2)],
                           'accuracy': [round(all_accuracy, 2), round(reduced_accuracy, 2)]})

comparison[['features', 'accuracy', 'run_time']]

relative_accuracy_decrease = 100 * (all_accuracy - reduced_accuracy) / all_accuracy
print('Relative decrease in accuracy:', round(relative_accuracy_decrease, 3), '%.')

relative_runtime_decrease = 100 * (all_features_time - reduced_features_time) / all_features_time
print('Relative decrease in run time:', round(relative_runtime_decrease, 3), '%.')<jupyter_output>Relative decrease in accuracy: 0.514 %.
Relative decrease in run time: 13.288 %.
{'fraction_non_alphanumeric': 0.11096833409821019, 'fraction_numerical': 0.02625057365764112, 'mean_word_length': 4.877022653721683, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '651621', 'license': 'MIT', 'n_tokens_mistral': 3624, 'n_tokens_neox': 3305, 'n_words': 1020, 'path': 'Boston House Price Prediction/Boston_house_prices_part1.ipynb', 'repo_name': 'theDeepanshuMourya/Machine-Learning-Projetcs', 'size': '10895'}