<jupyter_start><jupyter_text>DSCI 525 - Web and Cloud Computing Milestone 1: Tackling big data on your laptop Group 4 Members: <NAME>, <NAME>, <NAME>, <NAME> GitHub Repo: **https://github.com/UBC-MDS/group4-525**  Loading the libraries<jupyter_code>import re
import os
import glob
import zipfile
import requests
from urllib.request import urlretrieve
import json
import pandas as pd
from memory_profiler import memory_usage
import dask.dataframe as dd
import pyarrow.feather as feather
import pyarrow.dataset as ds
%load_ext rpy2.ipython
%load_ext memory_profiler<jupyter_output><empty_output><jupyter_text>Note: Code across this lab has been adapted from the DSCI-525 lectures  1. Downloading the data<jupyter_code># Necessary metadata
article_id = 14096681  
url = f"https://api.figshare.com/v2/articles/{article_id}"
headers = {"Content-Type": "application/json"}
output_directory = "figsharerainfall/"
response = requests.request("GET", url, headers=headers)
data = json.loads(response.text)  
files = data["files"]             
files<jupyter_output><empty_output><jupyter_text>2. Unzipping Data<jupyter_code>%%time
files_to_dl = ["data.zip"]  
for file in files:
    if file["name"] in files_to_dl:
        os.makedirs(output_directory, exist_ok=True)
        urlretrieve(file["download_url"], output_directory + file["name"])
%%time
with zipfile.ZipFile(os.path.join(output_directory, "data.zip"), 'r') as f:
    f.extractall(output_directory)<jupyter_output>CPU times: user 17.7 s, sys: 3.32 s, total: 21.1 s
Wall time: 21.5 s<jupyter_text>3. Combining data CSVs<jupyter_code>df = pd.read_csv("./figsharerainfall/ACCESS-CM2_daily_rainfall_NSW.csv")
df
%%time
%memit

#Merging all the csv files

files = glob.glob('figsharerainfall/*NSW.csv')
df = pd.concat((pd.read_csv(file, index_col=0)
                .assign(model=re.findall(r'/([^_]*)', file)[0])
                for file in files)
              )
df.to_csv("figsharerainfall/combined_data.csv")<jupyter_output>peak memory: 374.74 MiB, increment: 0.06 MiB
CPU times: user 6min 44s, sys: 23.1 s, total: 7min 7s
Wall time: 7min 15s<jupyter_text>The following table summerizes the `cpu times` and `Wall time` of combining csv files for all team members.|   |  Memory Usage |  CPU Time  | Wall Time  | OS  |  Memory (RAM) |  CPUs   ||---|---|---|---|---|---|---||  Heidi | 0.04 MiB  | 6min 45s  | 7min 10s  |  mac OS Catalina | 16 GB 3733 MHz LPDDR4X | 2 GHz Quad-Core Intel Core i5   || Junting  | 0.05 MiB  |  15min 8s | 15min 36s  | mac OS Catalina  |16GB of 2400MHz DDR4 | 2.6GHz 6-core Intel Core i7   || Kamal  |  0.79 MiB |  9min 20s  |  9min 43s | Windows 10 Pro  | 8 GB DDR4 SDRAM |  Intel Core i7-8650U Quad-Core  ||  Tanmay | 0.04 MiB  | 5min 23s  |  5min 28s |  mac OS Big Sur   | 16 GB 2667 MHz DDR4  | 2.6 GHz 6-Core Intel Core i7    |- We were able to combine the CSV files using Pandas concat method on both macOS and Windows operating systems and on the machines of all the 4 team-members.- `Memory usage` ranged from `0.04 MiB` to `0.79 MiB`, `CPU time` from `5min 23s` to `15min 8s` and Wall time from `5min 28s` to `15min 36s`. - Memory usage and CPU processing times are also impacted by background processes on the individual machines.<jupyter_code>%%sh
du -sh figsharerainfall/combined_data.csv<jupyter_output>5.6G	figsharerainfall/combined_data.csv<jupyter_text>4. Load the combined CSV to memory and perform a simple EDA<jupyter_code>%%time
%%memit
#loading the entire data to the memory using Pandas
df = pd.read_csv("figsharerainfall/combined_data.csv")
print(df["model"].value_counts())<jupyter_output>MPI-ESM1-2-HR       5154240
CMCC-ESM2           3541230
NorESM2-MM          3541230
TaiESM1             3541230
CMCC-CM2-SR5        3541230
CMCC-CM2-HR4        3541230
SAM0-UNICON         3541153
FGOALS-f3-L         3219300
GFDL-ESM4           3219300
GFDL-CM4            3219300
EC-Earth3-Veg-LR    3037320
MRI-ESM2-0          3037320
BCC-CSM2-MR         3035340
MIROC6              2070900
ACCESS-CM2          1932840
ACCESS-ESM1-5       1610700
INM-CM5-0           1609650
INM-CM4-8           1609650
FGOALS-g3           1287720
KIOST-ESM           1287720
MPI-ESM1-2-LR        966420
NESM3                966420
AWI-ESM-1-1-LR       966420
MPI-ESM-1-2-HAM      966420
NorESM2-LM           919800
CanESM5              551880
BCC-ESM1             551880
Name: model, dtype: int64
peak memory: 8403.29 MiB, increment: 4625.34 MiB
CPU times: user 1min 6s, sys: 13.3 s, total: 1min 19s
Wall time: 1min 22s<jupyter_text>|   |  Memory Usage |  CPU Time  | Wall Time  | OS  |  Memory (RAM) |  CPUs   ||---|---|---|---|---|---|---||  Heidi | 2933.94 MiB  | 54.9 s  | 1min 20s  |  mac OS Catalina | 16 GB 3733 MHz LPDDR4X | 2 GHz Quad-Core Intel Core i5   || Junting  | 4149.87 MiB  |  2min 54s | 3min 11s  | mac OS Catalina  |16GB of 2400MHz DDR4 | 2.6GHz 6-core Intel Core i7   || Kamal  |  1084.17 MiB |  4min 20s  |  4min 46s | Windows 10 Pro  | 8 GB DDR4 SDRAM |  Intel Core i7-8650U Quad-Core  ||  Tanmay | 3921.75 MiB  | 1min 1s  | 1min 4s  | mac OS Big Sur   | 16 GB 2667 MHz DDR4  | 2.6 GHz 6-Core Intel Core i7 |- We were able to load the combined CSV files using Pandas read_csv method on both macOS and Windows operating systems and on the machines of all the 4 team-members.- `Memory usage` ranged from `1084.17 MiB` to `4149.87 MiB`, `CPU time` from `54.9 s` to `4min 20s` and Wall time from `1min 4s` to `4min 46s`. - Memory usage and CPU processing times are also impacted by background processes on the individual machines.<jupyter_code>df.head()
#checking datatypes for columns
df.dtypes<jupyter_output><empty_output><jupyter_text>4.1. Investigate approaches to reduce memory usage while performing the EDA   4.1.1. Changing dtype of the data and loading just the columns we want<jupyter_code>print(f"Memory usage with float64: {df[['lat_min','lat_max','rain (mm/day)']].memory_usage().sum() / 1e6:.2f} MB")
print(f"Memory usage with float32: {df[['lat_min','lat_max','rain (mm/day)']].astype('float32', errors='ignore').memory_usage().sum() / 1e6:.2f} MB")<jupyter_output>Memory usage with float64: 1499.23 MB
Memory usage with float32: 749.61 MB<jupyter_text>4.1.2. Loading data in chunks using Pandas<jupyter_code>%%time
%%memit
counts = pd.Series(dtype=int)
for chunk in pd.read_csv("figsharerainfall/combined_data.csv", chunksize=10_000_000):
    counts = counts.add(chunk["model"].value_counts(), fill_value=0)
print(counts.astype(int))<jupyter_output>ACCESS-CM2          1932840
ACCESS-ESM1-5       1610700
AWI-ESM-1-1-LR       966420
BCC-CSM2-MR         3035340
BCC-ESM1             551880
CMCC-CM2-HR4        3541230
CMCC-CM2-SR5        3541230
CMCC-ESM2           3541230
CanESM5              551880
EC-Earth3-Veg-LR    3037320
FGOALS-f3-L         3219300
FGOALS-g3           1287720
GFDL-CM4            3219300
GFDL-ESM4           3219300
INM-CM4-8           1609650
INM-CM5-0           1609650
KIOST-ESM           1287720
MIROC6              2070900
MPI-ESM-1-2-HAM      966420
MPI-ESM1-2-HR       5154240
MPI-ESM1-2-LR        966420
MRI-ESM2-0          3037320
NESM3                966420
NorESM2-LM           919800
NorESM2-MM          3541230
SAM0-UNICON         3541153
TaiESM1             3541230
dtype: int64
peak memory: 6374.36 MiB, increment: 1354.38 MiB
CPU times: user 1min 6s, sys: 7.75 s, total: 1min 14s
Wall time: 1min 15s<jupyter_text>|   |  Memory Usage |  CPU Time  | Wall Time  | OS  |  Memory (RAM) |  CPUs   ||---|---|---|---|---|---|---||  Heidi | 5967.77 MiB | 53 s  | 1min 2s |  mac OS Catalina | 16 GB 3733 MHz LPDDR4X | 2 GHz Quad-Core Intel Core i5   || Junting  |2163.42  | 2min 18s | 2min 19s | mac OS Catalina  |16GB of 2400MHz DDR4 | 2.6GHz 6-core Intel Core i7   || Kamal  |  1619.86 MiB |  1min 54s  |  1min 59s | Windows 10 Pro  | 8 GB DDR4 SDRAM |  Intel Core i7-8650U Quad-Core  ||  Tanmay |  1271.02 MiB | 55.6 s  |  56.5 s | mac OS Big Sur   | 16 GB 2667 MHz DDR4  | 2.6 GHz 6-Core Intel Core i7  |- We were able to load data in chunks using Pandas on both macOS and Windows operating systems and on the machines of all the 4 team-members..- `Memory usage` ranged from `1271.02 MiB` to `5967.77 MiB`, `CPU time` from `53 s` to `2min 18s` and Wall time from `56.5 s` to `2min 19s`. - Memory usage and CPU processing times are also impacted by background processes on the individual machines.  4.1.2. Loading data using Dask<jupyter_code>%%time
%%memit
# Using dask
ddf = dd.read_csv('figsharerainfall/combined_data.csv')
print(ddf["model"].value_counts().compute())<jupyter_output>MPI-ESM1-2-HR       5154240
TaiESM1             3541230
NorESM2-MM          3541230
CMCC-CM2-HR4        3541230
CMCC-CM2-SR5        3541230
CMCC-ESM2           3541230
SAM0-UNICON         3541153
FGOALS-f3-L         3219300
GFDL-CM4            3219300
GFDL-ESM4           3219300
EC-Earth3-Veg-LR    3037320
MRI-ESM2-0          3037320
BCC-CSM2-MR         3035340
MIROC6              2070900
ACCESS-CM2          1932840
ACCESS-ESM1-5       1610700
INM-CM5-0           1609650
INM-CM4-8           1609650
KIOST-ESM           1287720
FGOALS-g3           1287720
MPI-ESM1-2-LR        966420
NESM3                966420
AWI-ESM-1-1-LR       966420
MPI-ESM-1-2-HAM      966420
NorESM2-LM           919800
BCC-ESM1             551880
CanESM5              551880
Name: model, dtype: int64
peak memory: 6610.67 MiB, increment: 2087.40 MiB
CPU times: user 1min 33s, sys: 24 s, total: 1min 57s
Wall time: 46 s<jupyter_text>|   |  Memory Usage |  CPU Time  | Wall Time  | OS  |  Memory (RAM) |  CPUs   ||---|---|---|---|---|---|---||  Heidi | 4714.38 MiB  | 1min 32s  | 42.6 s  |  mac OS Catalina | 16 GB 3733 MHz LPDDR4X | 2 GHz Quad-Core Intel Core i5   || Junting  | 1817.48 MiB  | 3min 43s | 1min 19s  | mac OS Catalina  |16GB of 2400MHz DDR4 | 2.6GHz 6-core Intel Core i7   || Kamal  |  1690.57 MiB|  2min 24s  |  1min 2s | Windows 10 Pro  | 8 GB DDR4 SDRAM |  Intel Core i7-8650U Quad-Core  ||  Tanmay |  1797.21 MiB | 1min 31s  |  34.3 s | mac OS Big Sur   | 16 GB 2667 MHz DDR4  | 2.6 GHz 6-Core Intel Core i7   |- We were able to load data using Dask on both macOS and Windows operating systems and on the machines of all the 4 team-members..- `Memory usage` ranged from `1690.57 MiB` to `4714.38 MiB`, `CPU time` from `1min 31s` to `3min 43s` and Wall time from `34.3 s` to `1min 19s`. - Memory usage and CPU processing times are also impacted by background processes on the individual machines.  4.2. Discuss your observations.We tried the following approaches to reduce the memory usage while performing the EDA:1. Changing dtype of the data and loading just the columns we want:    - We loaded only 3 out of the 5 attributes, namely, 'lat_min','lat_max', and 'rain (mm/day)'.    - We changed the data type of these attributes to float32 from the orignial float64.    - Memory usage with float32: 750.17 MB was almost half of the memory usage with float64: 1500.33 MB.    - This validates the hypothesis that using lower data types(float32 vs 64 in this case) leads to more efficient memory usage.    2. Loading data in chunks using Pandas    - We loaded the combined csv file using a chunksize=10_000_000 while performing the EDA.    - We observed a decline in peak memory usage from 8403.29 MiB to 6374.36 MiB.    - Wall Time decreased from 1min 22s to 1min 15s.    - We do not see a significant change in the Wall and CPU times.     - We hypothesize that this impact would be more pronoucned when doing more memory intensive operations in EDA and using smaller chunk sizes would further reduce the memory usage.    3. Loading data using Dask    - We next loaded the combined CSV using a dask object.     - We observed a decline in peak memory usage from 8403.29 MiB to 6610.67 MiB.    - Wall Time decreased significantly from 1min 22s to 46 s.    - We also notice that the CPU time was higher than the wall time (1min 33s vs 46s) suggesting that the CPU was performing operations in parallel. In summary, loading the entire data (combined_csv) to memory at once has the longest wall time and the highest memory usage as expected. We have looked at three different approaches to load the data more efficiently i.e. loading the entire data using pandas, loading the data in chunks, and loading data using Dask. We conclude that if we want to reduce the memory usage and the processing time, loading with Dask is the best option.  5. Perform a simple EDA in R<jupyter_code>%%time
%%memit

dataset = ds.dataset("figsharerainfall/combined_data.csv", format="csv")

table = dataset.to_table()
%%time
# writing in feather format 
feather.write_feather(table, 'figsharerainfall/combined_data.feather')
%%time
%%R

library(arrow)
start_time <- Sys.time()
r_table <- arrow::read_feather("figsharerainfall/combined_data.feather")
print(class(r_table))
library(dplyr)
result <- r_table %>% count(model)
end_time <- Sys.time()
print(result)
print(end_time - start_time)
%%R

library(tidyverse)

r_table <- r_table %>% rename(rain_mmperday = `rain (mm/day)`)

summary_table <- r_table %>%
    drop_na() %>%
    summarise(median_lat_min = median(lat_min),
             median_lat_max = median(lat_max),
             median_lon_min = median(lon_min),
             median_lon_max = median(lon_max),
             median_rain = median(rain_mmperday))
    
summary_table<jupyter_output>R[write to console]: ── [1mAttaching packages[22m ─────────────────────────────────────── tidyverse 1.3.0 ──

R[write to console]: [32m✔[39m [34mggplot2[39m 3.3.3     [32m✔[39m [34mpurrr  [39m 0.3.4
[32m✔[39m [34mtibble [39m 3.1.0     [32m✔[39m [34mstringr[39m 1.4.0
[32m✔[39m [34mtidyr  [39m 1.1.3     [32m✔[39m [34mforcats[39m 0.5.1
[32m✔[39m [34mreadr  [39m 1.4.0     

R[write to console]: ── [1mConflicts[22m ────────────────────────────────────────── tidyverse_conflicts() ──
[31m✖[39m [34mdplyr[39m::[32mfilter()[39m masks [34mstats[39m::filter()
[31m✖[39m [34mdplyr[39m::[32mlag()[39m    masks [34mstats[39m::lag()
{'fraction_non_alphanumeric': 0.10944098489728724, 'fraction_numerical': 0.10478849044449216, 'mean_word_length': 3.0545560069645967, 'pattern_counts': {'":': 1, '<': 47, '<?xml version=': 0, '>': 45, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '146580', 'license': 'MIT', 'n_tokens_mistral': 6054, 'n_tokens_neox': 4964, 'n_words': 1608, 'path': 'notebooks/milestone1.ipynb', 'repo_name': 'UBC-MDS/group4-525', 'size': '13998'}