<jupyter_start><jupyter_text>Milestone 2 Link to github repo for this milestone: https://github.com/UBC-MDS/DSCI525-Group7/blob/main/notebooks/Milestone2.ipynb  Milestone 2 checklist  You will have mainly 2 tasks. Here is the checklist...- To set up a collaborative environment     - Setup your EC2 instance with JupyterHub.    - Install all necessary things needed in your UNIX server (amazon ec2 instance).    - Set up your S3 bucket.    - Move the data that you wrangled in your last milestone to s3.    - To move data from s3.- Wrangle the data in preparation for machine learning    - Get the data from S3 in your notebook and make data ready for machine learning.  1. Setup your EC2 instance  Screenshot 1 (Set up EC2 instance):![](https://github.com/UBC-MDS/DSCI525-Group7/blob/main/img/Milestone_2/screenshot1_instance.png?raw=true)  2. Setup your JupyterHub  Screenshot 2 (Set up JupyterHub):![](https://github.com/UBC-MDS/DSCI525-Group7/blob/main/img/Milestone_2/screenshot2_jupyter.png?raw=true)  3. Setup the server   Screenshot 3 (Set up server):![](https://github.com/UBC-MDS/DSCI525-Group7/blob/main/img/Milestone_2/screenshot3_server.png?raw=true)IP address is masked.  4. Get the data what we wrangled in our first milestone.  You have to install the packages that are needed. Refer this TLJH [document]( https://tljh.jupyter.org/en/latest/howto/env/user-environment.html).Refer ```pip``` section.Don't forget to add option -E. This way, all packages that you install will be available to other users in your JupyterHub.These packages you must install and install other packages needed for your wrangling.    sudo -E pip install pandas    sudo -E pip install pyarrow    sudo -E pip install s3fs As in the last milestone, we looked at getting the data transferred from Python to R, and we have different solutions. Henceforth, I uploaded the parquet file format, which we can use moving forward.<jupyter_code>import re
import os
import glob
import zipfile
import requests
from urllib.request import urlretrieve
import json
import pandas as pd<jupyter_output><empty_output><jupyter_text>Remember here we gave the folder that we created in Step 3.2 as we made it available for all the users in a group.<jupyter_code># Necessary metadata
article_id = 14226968  # this is the unique identifier of the article on figshare
url = f"https://api.figshare.com/v2/articles/{article_id}"
headers = {"Content-Type": "application/json"}
output_directory = "/srv/data/my_shared_data_folder/"
response = requests.request("GET", url, headers=headers)
data = json.loads(response.text)  # this contains all the articles data, feel free to check it out
files = data["files"]             # this is just the data about the files, which is what we want
files
files_to_dl = ["combined_model_data_parti.parquet"]  ## Please download the partitioned 
for file in files:
    if file["name"] in files_to_dl:
        os.makedirs(output_directory, exist_ok=True)
        urlretrieve(file["download_url"], output_directory + file["name"])<jupyter_output><empty_output><jupyter_text>5. Setup your S3 bucket and move data  Screenshot 4 (Set up S3 bucket):![image.png](attachment:ac1852b4-b5d6-4621-9b05-de0ac54d81eb.png)  6. Wrangle the data in preparation for machine learning Our data currently covers all of NSW, but say that our client wants us to create a machine learning model to predict rainfall over Sydney only. There's a bit of wrangling that needs to be done for that:1. We need to query our data for only the rows that contain information covering Sydney2. We need to wrangle our data into a format suitable for training a machine learning model. That will require pivoting, resampling, grouping, etc.To train an ML algorithm we need it to look like this:||model-1_rainfall|model-2_rainfall|model-3_rainfall|...|observed_rainfall||---|---|---|---|---|---||0|0.12|0.43|0.35|...|0.31||1|1.22|0.91|1.68|...|1.34||2|0.68|0.29|0.41|...|0.57| 6.1) Get the data from s3 (```combined_model_data_parti.parquet``` and ```observed_daily_rainfall_SYD.csv```)6.2) First query for Sydney data and then drop the lat and lon columns (we don't need them).```syd_lat = -33.86syd_lon = 151.21```Expected shape ```(1150049, 2)```.6.3) Save this processed file to s3 for later use:  Save as a csv file ```ml_data_SYD.csv``` to ```s3://mds-s3-xxx/output/```  expected shape ```(46020,26)``` - This includes all the models as columns and also adding additional column ```Observed``` loaded from ```observed_daily_rainfall_SYD.csv``` from s3.<jupyter_code>import pandas as pd
import pyarrow.parquet as pq
#pip install 's3fs'
# Pass AWS credentials and read in data
aws_credentials ={"key": "",
                  "secret": "",
                  "token":""} 

df = pd.read_parquet("s3://mds-s3-group7/combined_data_partitioned.parquet",
                    storage_options = aws_credentials)
df.head()
df.head()
df.shape
# Read in data from `observed rainfall` dataframe
df_observed = pd.read_csv("s3://mds-s3-group7/observed_daily_rainfall_SYD.csv", 
                          storage_options=aws_credentials)
df_observed["model"] = "observed"
df_observed
# Select data with only Sydney lat and lon
df = df.query("lat_min <= -33.86 and lat_max >= -33.86 and lon_min <= 151.21 and lon_max >= 151.21")
df.head()
df.shape
# Drop lat and lon columns
df = df.drop(columns = ["lat_min", "lat_max", "lon_min", "lon_max"])
df.head()
# Combine Sydney data with observed rainfall
df = pd.concat([df, df_observed])
df
# Remove time in the `time` column

df['time'] = pd.to_datetime(df["time"]).dt.date
df.head()
# Set index to column `time`
df.set_index("time", inplace=True)
df
# Pivot dataframe to wide format displaying models in columns
df_output = df.reset_index().pivot(values="rain (mm/day)", columns="model", index="time")
df_output
df_output.shape
# Save processed file to csv
df_output.to_csv("s3://mds-s3-group7/output/ml_data_SYD.csv", 
                 storage_options = aws_credentials)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11101836393989983, 'fraction_numerical': 0.030717863105175294, 'mean_word_length': 4.3111702127659575, 'pattern_counts': {'":': 4, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '607975', 'license': 'MIT', 'n_tokens_mistral': 2018, 'n_tokens_neox': 1876, 'n_words': 653, 'path': 'notebooks/Milestone2.ipynb', 'repo_name': 'UBC-MDS/DSCI525-Group7', 'size': '5990'}