<jupyter_start><jupyter_text>Milestone 3 Group 18 DSCI 525 -  Web and Cloud Computing  Task 3  Developing a machie learning model using scikit-learn for the observations of rainfall data in Sydney, Australis.  Library Imports<jupyter_code>import numpy as np
import pandas as pd
from joblib import dump, load
from sklearn.metrics import mean_squared_error
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import FeatureUnion, Pipeline, make_pipeline
plt.style.use('ggplot')
plt.rcParams.update({'font.size': 16, 'axes.labelweight': 'bold', 'figure.figsize': (8,6)})
from memory_profiler import memory_usage
from pandas_profiling import ProfileReport 
%load_ext memory_profiler<jupyter_output><empty_output><jupyter_text>Part 1: Recall as a final goal of this project. We want to build and deploy ensemble machine learning models in the cloud, where features are outputs of different climate models and the target is the actual rainfall observation. In this milestone, you'll actually build these ensemble machine learning models in the cloud.  **Your tasks:**1. Read the data CSV from your s3 bucket. 2. Drop rows with nans. 3. Split the data into train (80%) and test (20%) portions with `random_state=123`. 4. Carry out EDA of your choice on the train split. 5. Train ensemble machine learning model using `RandomForestRegressor` and evaluate with metric of your choice (e.g., `RMSE`) by considering `Observed` as the target column. 6. Discuss your results. Are you getting better results with ensemble models compared to the individual climate models? > Recall that individual columns in the data are predictions of different climate models.   1.1 Read the data CSV from your s3 bucket.<jupyter_code>%%time
%%memit

## You could download it from your bucket, or you can use the file that I have in my bucket. 
## You should be able to access it from my bucket using your key and secret
aws_credentials ={"key": "<enter your secret here>","secret": "<enter your secret here>"} ## dont include you secret and key when submitting the notebook
df = pd.read_csv("s3://mds-s3-student65/output/ml_data_SYD.csv", storage_options=aws_credentials, index_col = "time")<jupyter_output>peak memory: 238.02 MiB, increment: 88.93 MiB
CPU times: user 921 ms, sys: 225 ms, total: 1.15 s
Wall time: 2.02 s<jupyter_text>Displaying some of the dataframe rows as well as the shape of the dataframe.<jupyter_code>df
df.shape<jupyter_output><empty_output><jupyter_text>1.2 Dropping rows with nans.<jupyter_code>df.dropna(inplace = True)
df
df.shape<jupyter_output><empty_output><jupyter_text>We see that we have dropped rows in the original dataframe that had missing values.  1.3 Split the data into train (80%) and test (20%) portions with random_state=123<jupyter_code>df_train, df_test =  train_test_split(df, test_size =0.2, random_state = 123)
df_train.shape
df_test.shape<jupyter_output><empty_output><jupyter_text>1.4 Carry out EDA of your choice on the train split.  First let us check the dtypes of the columns in the train dataframe.<jupyter_code>df_train.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Index: 36791 entries, 1953-10-26 to 1932-01-31
Data columns (total 26 columns):
 #   Column            Non-Null Count  Dtype  
---  ------            --------------  -----  
 0   ACCESS-CM2        36791 non-null  float64
 1   ACCESS-ESM1-5     36791 non-null  float64
 2   AWI-ESM-1-1-LR    36791 non-null  float64
 3   BCC-CSM2-MR       36791 non-null  float64
 4   BCC-ESM1          36791 non-null  float64
 5   CMCC-CM2-HR4      36791 non-null  float64
 6   CMCC-CM2-SR5      36791 non-null  float64
 7   CMCC-ESM2         36791 non-null  float64
 8   CanESM5           36791 non-null  float64
 9   EC-Earth3-Veg-LR  36791 non-null  float64
 10  FGOALS-g3         36791 non-null  float64
 11  GFDL-CM4          36791 non-null  float64
 12  INM-CM4-8         36791 non-null  float64
 13  INM-CM5-0         36791 non-null  float64
 14  KIOST-ESM         36791 non-null  float64
 15  MIROC6            36791 non-null  float64
 16  MPI-ESM-1-2-HAM   36791 non-nul[...]<jupyter_text>All of the columns in the train_df are of type numeric (float64). Now let us consider statistics of the numerical columns in the dataframe with `describe()`.<jupyter_code>df_train.describe()
profile = ProfileReport(df_train, minimal=True)
profile.to_file("output.html")<jupyter_output><empty_output><jupyter_text>For more elaborate EDA, please check `output.html` file for pandas-profiling report.  1.5 Train ensemble machine learning model using RandomForestRegressor and evaluate with metric of your choice (e.g., RMSE) by considering Observed as the target column.<jupyter_code>X_train, y_train = df_train.drop(columns=["observed"]), df_train["observed"]
X_test, y_test = df_test.drop(columns=["observed"]), df_test["observed"]<jupyter_output><empty_output><jupyter_text>Now let's train X_train with `RandomForestRegressor` model with default hyperparameters.<jupyter_code>%%time
%%memit
model = RandomForestRegressor()
model.fit(X_train, y_train)<jupyter_output>peak memory: 504.30 MiB, increment: 268.21 MiB
CPU times: user 3min 10s, sys: 217 ms, total: 3min 10s
Wall time: 3min 10s<jupyter_text>Now let's use the trained model to make predictions on the test data, and compare it with the actual observed values for the rainfall amount. **Training Accuracy:** Let's check first the training accuracy of the trained `RandomForestRegressor` model.<jupyter_code>mean_squared_error(y_train, model.predict(X_train), squared=False)<jupyter_output><empty_output><jupyter_text>**Test Accuracy**: Now let us compute the test accuracy, by using the model to predict on unseen observations.<jupyter_code>rmse = mean_squared_error(y_test, model.predict(X_test), squared=False)
rmse<jupyter_output><empty_output><jupyter_text>1.6 Discuss your results. Are you getting better results with ensemble models compared to the individual climate models?  Yes, we expect to get better results in comparison to individual climate models, sicne we are ensembling different climate models in order to use the information from this ensembled methods to make prediction for the actual observed amount of rainfall.We can further evaluate this by comparing the forecasts of rainfall amount from a particular model with actual observed rainfall amount.<jupyter_code>mean_squared_error(y_train, X_train["ACCESS-CM2"], squared=False)<jupyter_output><empty_output><jupyter_text>We see that the RMSE of the prediction of the model `"ACCESS-CM2` with the actual observed values is higher than the RMSE with the predicion of the ensemble model and actual observed values. Let's look at the RMSE for the predictions of the remaining models and the actual observed rainfall.<jupyter_code>columns = X_train.columns
columns
result={}
for col in columns:
    rmse = mean_squared_error(y_train, X_train[col], squared=False)
    result[col] = rmse
result<jupyter_output><empty_output><jupyter_text>We see that none of the individual models performs better compared to the ensemble approach using the infromation from the individual models.  Part 2:  Preparation for deploying model next week  Completed task 4 from the milestone3 . We’ve found ```n_estimators=100, max_depth=5``` to be the best hyperparameter settings with MLlib (from the task 4 from milestone3), here we then use the same hyperparameters to train a scikit-learn model.<jupyter_code>%%time
%%memit
model = RandomForestRegressor(n_estimators=100, max_depth=5, bootstrap =True)
model.fit(X_train, y_train)<jupyter_output>peak memory: 505.63 MiB, increment: 0.00 MiB
CPU times: user 35.3 s, sys: 19.9 ms, total: 35.3 s
Wall time: 35.5 s<jupyter_text>Now let's use the trained `RandomForestRegressor` model with the optimized hyperparameters to forecast observed rainfall for the test data and compute the metric `rmse`.<jupyter_code>print(f"Train RMSE: {mean_squared_error(y_train, model.predict(X_train), squared=False):.2f}")
print(f" Test RMSE: {mean_squared_error(y_test, model.predict(X_test), squared=False):.2f}")
# ready to deploy
dump(model, "model.joblib")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08065296251511488, 'fraction_numerical': 0.039177750906892385, 'mean_word_length': 4.937544867193108, 'pattern_counts': {'":': 2, '<': 60, '<?xml version=': 0, '>': 61, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '137327', 'license': 'MIT', 'n_tokens_mistral': 2797, 'n_tokens_neox': 2500, 'n_words': 939, 'path': 'notebooks/Milestone3/Milestone3-Task3.ipynb', 'repo_name': 'UBC-MDS/525_group18', 'size': '8270'}