<jupyter_start><jupyter_text>Task 3  Imports<jupyter_code>import json
import numpy as np
import pandas as pd
from joblib import dump, load

from sklearn.metrics import mean_squared_error
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split

import matplotlib.pyplot as plt
import plotly.express as px

plt.style.use('ggplot')
plt.rcParams.update({'font.size': 16, 'axes.labelweight': 'bold', 'figure.figsize': (8,6)})

from statsmodels.tsa.api import seasonal_decompose
import matplotlib as mpl

import joblib<jupyter_output>Matplotlib created a temporary config/cache directory at /tmp/matplotlib-uep9b9mt because the default path (/home/jupyter-student34/.cache/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.<jupyter_text>Part 1: Recall as a final goal of this project. We want to build and deploy ensemble machine learning models in the cloud, where features are outputs of different climate models and the target is the actual rainfall observation. In this milestone, you'll actually build these ensemble machine learning models in the cloud.  **Your tasks:**1. Read the data CSV from your s3 bucket. 2. Drop rows with nans. 3. Split the data into train (80%) and test (20%) portions with `random_state=123`. 4. Carry out EDA of your choice on the train split. 5. Train ensemble machine learning model using `RandomForestRegressor` and evaluate with metric of your choice (e.g., `RMSE`) by considering `Observed` as the target column. 6. Discuss your results. Are you getting better results with ensemble models compared to the individual climate models? > Recall that individual columns in the data are predictions of different climate models.   Step 1: Read data<jupyter_code>## You could download it from your bucket, or you can use the file that I have in my bucket. 
## You should be able to access it from my bucket using your key and secret
# aws_credentials ={"key": "","secret": ""} 

# Credential is stored in a file to avoid accidential commit
f = open("keys.txt", "r")
aws_credentials = json.loads(f.read())
df = pd.read_csv("s3://mds-s3-student34/output/ml_data_SYD.csv", index_col=0, parse_dates=True, storage_options=aws_credentials)<jupyter_output><empty_output><jupyter_text>Step 2: Drop rows with nans<jupyter_code>df = df.dropna()<jupyter_output><empty_output><jupyter_text>Step 3: Split data<jupyter_code>train_df, test_df = train_test_split(df, test_size=0.2, random_state=123, shuffle=False) # we are dealing with timeseries hence avoiding shuffling the data when splitting, although it should not matter much in this case<jupyter_output><empty_output><jupyter_text>Step 4: EDA<jupyter_code># EDA
train_df["Observed"].plot.line(xlabel="Time", ylabel="Rainfall", legend=False, figsize=(20,5));
# resample the data into months to have better view of trend and seasonality
monthly_data = train_df[["Observed"]].resample("1M").mean()
monthly_data.index.fred="M"
monthly_data.plot();
# decompose the timeseries into trend and seasonality
model = seasonal_decompose(monthly_data[["Observed"]], model="additive", period=12) 
with mpl.rc_context():  
    mpl.rc("figure", figsize=(7, 15))
    model.plot()
    plt.tight_layout()
train_df.describe()<jupyter_output><empty_output><jupyter_text>**Highlights:**- There is seasonality in the timeseries for every 12 months, which makes sense- There is no trend detected in the data- The prediction varies considerably among models  Step 5: Train ensemble machine learning model using RandomForestRegressor<jupyter_code>X_train, y_train = train_df.drop(columns="Observed"), train_df["Observed"]
X_test, y_test = test_df.drop(columns="Observed"), test_df["Observed"]
# Train ensemble ML model using `RandomForestRegressor`
import os

test_mode = False # set test_mode to True to avoid re-training the model during testing stage
if (test_mode and os.path.exists("first_model.joblib")):
    model = joblib.load("first_model.joblib")
else:
    model = RandomForestRegressor(random_state=123)
    model.fit(X_train, y_train)
    dump(model, "first_model.joblib") # save model so that we can reload during testing
# Evaluate
print(f"Train RMSE: {mean_squared_error(y_train, model.predict(X_train), squared=False):.2f}")
print(f" Test RMSE: {mean_squared_error(y_test, model.predict(X_test), squared=False):.2f}")<jupyter_output>Train RMSE: 3.24
 Test RMSE: 8.08<jupyter_text>Step 6: Results discussion<jupyter_code># create a column to store the prediction of the ensemble model
test_df = test_df.assign(Ensemble = model.predict(X_test))
test_df.head()
# visualize the predictions of all models compared to the observed data
# there are too many models to have a clear plot hence disabling this code
# r_test_df = test_df.reset_index()
# result_df = pd.melt(r_test_df, value_vars = r_test_df.columns[1:], 
#                     var_name="Model", value_name="Rainfall", id_vars = ["time"]).set_index("time")
# px.line(result_df, y="Rainfall", color="Model", width=800)
# create a data frame to store MSE of all models compared to the observed data
mse_results = {}
cols = test_df.columns.to_list() # get the list of all model names
cols.remove("Observed")
for c in cols:
    mse_results[c] = mean_squared_error(test_df["Observed"], test_df[c], squared=False) # store MSE of the current model to the dictionary
mse_results_df = pd.DataFrame.from_dict(mse_results, orient="index", columns=["MSE"]).sort_values(by="MSE")
mse_results_df<jupyter_output><empty_output><jupyter_text>> results discussion -  Are you getting better results with ensemble models compared to the individual climate models? **Comments**: Ensemble machine learning model using RandomForestRegressor generates better predictions in terms of lower mean squared errors in comparison to the individual climate models.  Part 2:  Preparation for deploying model next week We’ve found ```n_estimators=100, max_depth=5``` to be the best hyperparameter settings with MLlib (from the task 4 from milestone3), here we then use the same hyperparameters to train a scikit-learn model.<jupyter_code>model = RandomForestRegressor(n_estimators=100, max_depth=5) 
model.fit(X_train, y_train)
print(f"Train RMSE: {mean_squared_error(y_train, model.predict(X_train), squared=False):.2f}")
print(f" Test RMSE: {mean_squared_error(y_test, model.predict(X_test), squared=False):.2f}")
# ready to deploy
dump(model, "model.joblib")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09661762475352646, 'fraction_numerical': 0.011982405581677538, 'mean_word_length': 6.128648648648649, 'pattern_counts': {'":': 2, '<': 31, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '495310', 'license': 'MIT', 'n_tokens_mistral': 2113, 'n_tokens_neox': 2008, 'n_words': 691, 'path': 'notebooks/Milestone3-Task3.ipynb', 'repo_name': 'UBC-MDS/525_group11', 'size': '6593'}