<jupyter_start><jupyter_text>d-sandbox    Applied Random Forest Lab**Objective**: *Apply random forests to a regression problem in an effort to improve model generalization.*In this lab you will complete a series of guided exercises where you will build a random forest model to solve a regression problem. You will need to prepare the categorical variable appropriately and assess the output of the model. When complete, please use your answers to the exercises to answer questions in the following quiz within Coursera.<jupyter_code>%run ../../Includes/Classroom-Setup<jupyter_output><empty_output><jupyter_text>Exercise 1In this exercise, you will use the user-level lifestyle table. Run the following cell to make sure you can access the `adsda.ht_user_metrics_lifestyle` table.<jupyter_code>%sql
SELECT *
FROM adsda.ht_user_metrics_lifestyle
LIMIT 10<jupyter_output><empty_output><jupyter_text>Fill in the following cell to create a Pandas DataFrame from the Spark table.<jupyter_code># TODO
ht_metrics_pd_df = spark.table("adsda.ht_user_metrics_lifestyle").toPandas()<jupyter_output><empty_output><jupyter_text>Exercise 2In this exercise, you will encode the categorical feature `lifestyle` column using `LabelEncoder`.Fill in the blanks to complete this task.<jupyter_code># TODO
from sklearn.preprocessing import LabelEncoder

le = LabelEncoder()
ht_metrics_pd_df['lifestyle_cat'] = le.fit_transform(ht_metrics_pd_df["lifestyle"])
ht_metrics_pd_df['lifestyle_cat'].unique()
ht_metrics_pd_df.head()<jupyter_output><empty_output><jupyter_text>Exercise 3 In this exercise, you will build a random forest regression modelWe will once again try to predict a user's average `vo2` using their other metrics.Remember to set the `random_state` to 42! Before splitting the data and fitting the model, import the packages you will need from sklearn for the train test split and the Random Forest regressor.<jupyter_code># TODO
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
# TODO
X = ht_metrics_pd_df[['avg_resting_heartrate', 'avg_active_heartrate', 'avg_bmi', 'avg_steps', 'avg_workout_minutes', 'lifestyle_cat']]
y = ht_metrics_pd_df['avg_vo2']

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42) 

rf = RandomForestRegressor(random_state=42)
rf.fit(X_train, y_train)

y_train_predicted = rf.predict(X_train)
y_test_predicted = rf.predict(X_test)

print("R2 on training set: ", round(rf.score(X_train, y_train), 3))
print("R2 on test set: ", round(rf.score(X_test, y_test), 3))<jupyter_output><empty_output><jupyter_text>**Coursera Quiz:** For the `rf` model, what is the R-squared score on the training and test set?  Exercise 4Even though the untuned random forest did very well already, explore how tuning some hyperparameters affects the output.You will build three models:1. With `n_estimators`=101. With `max_depth`=21. With `bootstrap`=False<jupyter_code>rf_tuned_1 = RandomForestRegressor(n_estimators=10, random_state=42, bootstrap=False)

rf_tuned_1.fit(X_train, y_train)

y_train_predicted = rf_tuned_1.predict(X_train)
y_test_predicted = rf_tuned_1.predict(X_test)

print("R2 on training set: ", round(rf_tuned_1.score(X_train, y_train),3))
print("R2 on test set: ", round(rf_tuned_1.score(X_test, y_test), 3))<jupyter_output><empty_output><jupyter_text>**Coursera Quiz:** For the `rf_tuned_1` model, what is the R-squared score on the training and test set?<jupyter_code>rf_tuned_2 = RandomForestRegressor(max_depth=2, random_state=42)

rf_tuned_2.fit(X_train, y_train)

y_train_predicted = rf_tuned_2.predict(X_train)
y_test_predicted = rf_tuned_2.predict(X_test)

print("R2 on training set: ", round(rf_tuned_2.score(X_train, y_train),3))
print("R2 on test set: ", round(rf_tuned_2.score(X_test, y_test), 3))<jupyter_output><empty_output><jupyter_text>**Coursera Quiz:** For the `rf_tuned_2` model, what is the R-squared score on the training and test set?<jupyter_code>rf_tuned_3 = RandomForestRegressor(bootstrap=False, random_state=42)

rf_tuned_3.fit(X_train, y_train)

y_train_predicted = rf_tuned_3.predict(X_train)
y_test_predicted = rf_tuned_3.predict(X_test)

print("R2 on training set: ", round(rf_tuned_3.score(X_train, y_train),3))
print("R2 on test set: ", round(rf_tuned_3.score(X_test, y_test), 3))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0956221198156682, 'fraction_numerical': 0.014976958525345621, 'mean_word_length': 7.596039603960396, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '410380', 'license': 'MIT', 'n_tokens_mistral': 1581, 'n_tokens_neox': 1494, 'n_words': 399, 'path': 'Applied Data Science for Data Analysts/ipynb/4.2.2 Lab - Applied Random Forest.ipynb', 'repo_name': 'bsimmooes/Data-Science-with-Databricks-for-Data-Analysts-Coursera', 'size': '4340'}