<jupyter_start><jupyter_text>Task 3 **Group 7's submission for Milestone 3**  Imports<jupyter_code>import numpy as np
import pandas as pd
from joblib import dump, load
from sklearn.metrics import mean_squared_error
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
plt.style.use('ggplot')
plt.rcParams.update({'font.size': 16, 'axes.labelweight': 'bold', 'figure.figsize': (8,6)})
from sklearn.model_selection import (
    GridSearchCV,
    RandomizedSearchCV,
    ShuffleSplit,
    cross_val_score,
    cross_validate,
    train_test_split,
)

from sklearn.metrics import mean_squared_error<jupyter_output><empty_output><jupyter_text>Part 1: Recall as a final goal of this project. We want to build and deploy ensemble machine learning models in the cloud, where features are outputs of different climate models and the target is the actual rainfall observation. In this milestone, you'll actually build these ensemble machine learning models in the cloud.  **Your tasks:**1. Read the data CSV from your s3 bucket. 2. Drop rows with nans. 3. Split the data into train (80%) and test (20%) portions with `random_state=123`. 4. Carry out EDA of your choice on the train split. 5. Train ensemble machine learning model using `RandomForestRegressor` and evaluate with metric of your choice (e.g., `RMSE`) by considering `Observed` as the target column. 6. Discuss your results. Are you getting better results with ensemble models compared to the individual climate models? > Recall that individual columns in the data are predictions of different climate models.<jupyter_code>## Use your ML skills to get from step 1 to step 6
# 1) Read the data CSV from your s3 bucket
## You could download it from your bucket, or you can use the file that I have in my bucket. 
## You should be able to access it from my bucket using your key and secret
aws_credentials ={"key": "<your key>","secret": "<your secret>"} 
df = pd.read_csv("s3://mds-s3-student74/output/ml_data_SYD.csv", index_col=0, parse_dates=True, storage_options=aws_credentials)
df.head()
df.shape
# 2) Drop rows with nans
df = df.dropna()
df.shape
# 3) Split the data into train (80%) and test (20%) portions with random_state=123.
train_df, test_df = train_test_split(df, test_size = 0.2, random_state = 123)
# 4) Carry out EDA of your choice on the train split.
train_df.describe()<jupyter_output><empty_output><jupyter_text>**Comment 1:** The distributions for both modelled and observed rainfall data are highly left-skewed. The modelled and observed data have different mean and mediance values ranging from 0 to 4 mm/day. Observing the max value of the respective columns in `train_df`, there are upperbound outliers.<jupyter_code>plt.boxplot(train_df, labels = train_df.columns, vert=False, showmeans = True, showfliers = False);
plt.subplots_adjust(left=0.25)
plt.show();<jupyter_output><empty_output><jupyter_text>**Comment 2:** There seems to be a spike in daily rainfall for dungin the 1990s for observed data and no single model can capture this. The observed data also have high outliers. However, the overall pattern (except for the outliers) are relatively similar, we can employ the prediction power of essembling all these models.<jupyter_code>plt.rcParams.update({'font.size': 16, 'axes.labelweight': 'bold', 'figure.figsize': (20,20)})
train_df.plot(subplots = True, layout=(7,4));
plt.show();
# 5) Train ensemble machine learning model using RandomForestRegressor and 
# evaluate with metric of your choice (e.g., RMSE) by considering Observed as the target column.

X_train = train_df.drop(columns =['Observed'])
y_train = train_df['Observed']

model = RandomForestRegressor(n_jobs = -1, random_state = 100, max_depth = 10)
model.fit(X_train, y_train)
# Helper function for cross valuation
# Attribution: Varada's DSCI_573 lecture notes

def mean_std_cross_val_scores(model, X_train, y_train, **kwargs):
    """
    Returns mean and std of cross validation
    """
    scores = cross_validate(model, X_train, y_train, **kwargs)

    mean_scores = pd.DataFrame(scores).mean()
    std_scores = pd.DataFrame(scores).std()
    out_col = []

    for i in range(len(mean_scores)):
        out_col.append((f"%0.3f (+/- %0.3f)" % (mean_scores[i], std_scores[i])))

    return pd.Series(data=out_col, index=mean_scores.index)
scoring_metric = "neg_root_mean_squared_error"
results = {}
results["Random forest"] = mean_std_cross_val_scores(
    model, X_train, y_train,  n_jobs = -1, return_train_score=True, scoring=scoring_metric
)

pd.DataFrame(results)
X_test = test_df.drop(columns =['Observed'])
y_test = test_df['Observed']
y_pred = model.predict(X_test) # obtain predictions from test set
predictions = X_test.copy()
predictions['Ensemble'] = y_pred.tolist() # Add ensemble model results in prediction dataframe
predictions
# 6) Discuss your results. Are you getting better results with ensemble models compared to the individual climate models?
scores = {}
for col in predictions.columns:
    scores[col] = mean_squared_error(y_test, predictions[col], squared=False)*(-1)
    
pd.DataFrame(scores, index = ['Negative RMSE']).T<jupyter_output><empty_output><jupyter_text>**Comment:** By essembling the results of the 25 models, we managed to obtained better results with lower error (i.e. 8.6) on the test set as compared to individual climate models.  Part 2:  Preparation for deploying model next week  Complete task 4 from the milestone3 before coming here We’ve found ```n_estimators=100, max_depth=5``` to be the best hyperparameter settings with MLlib (from the task 4 from milestone3), here we then use the same hyperparameters to train a scikit-learn model.<jupyter_code>model = RandomForestRegressor(n_estimators=100, max_depth=5)
model.fit(X_train, y_train)
print(f"Train RMSE: {mean_squared_error(y_train, model.predict(X_train), squared=False):.2f}")
print(f" Test RMSE: {mean_squared_error(y_test, model.predict(X_test), squared=False):.2f}")
# ready to deploy
dump(model, "model.joblib")<jupyter_output><empty_output><jupyter_text>***Upload model.joblib to s3. You choose how you want to upload it.***<jupyter_code># Attribution: https://stackoverflow.com/questions/15085864/how-to-upload-a-file-to-directory-in-s3-bucket-using-boto

import boto3

session = boto3.Session(
    aws_access_key_id = aws_credentials['key'],
    aws_secret_access_key = aws_credentials['secret'],
)
s3 = session.resource('s3')
# Filename - File to upload
# Bucket - Bucket to upload to (the top level directory under AWS S3)
# Key - S3 object name (can contain subdirectories). If not specified then file_name is used
s3.meta.client.upload_file(Filename='model.joblib', Bucket='mds-s3-student74', Key='model.joblib')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10061782877316858, 'fraction_numerical': 0.01779935275080906, 'mean_word_length': 5.50622009569378, 'pattern_counts': {'":': 2, '<': 27, '<?xml version=': 0, '>': 28, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '345494', 'license': 'MIT', 'n_tokens_mistral': 2230, 'n_tokens_neox': 2074, 'n_words': 728, 'path': 'notebooks/Milestone3-Task3.ipynb', 'repo_name': 'UBC-MDS/525_group7', 'size': '6798'}