<jupyter_start><jupyter_text>Loading the combined CSV to memory<jupyter_code>import pandas as pd
import dask.dataframe as dd
import numpy as np
import matplotlib.pyplot as plt

%load_ext rpy2.ipython
%load_ext memory_profiler
!pip install matplotlib<jupyter_output>Requirement already satisfied: matplotlib in c:\users\vignesh\miniconda3\envs\525\lib\site-packages (3.4.1)
Requirement already satisfied: cycler>=0.10 in c:\users\vignesh\miniconda3\envs\525\lib\site-packages (from matplotlib) (0.10.0)
Requirement already satisfied: kiwisolver>=1.0.1 in c:\users\vignesh\miniconda3\envs\525\lib\site-packages (from matplotlib) (1.3.1)
Requirement already satisfied: pillow>=6.2.0 in c:\users\vignesh\miniconda3\envs\525\lib\site-packages (from matplotlib) (8.1.2)
Requirement already satisfied: python-dateutil>=2.7 in c:\users\vignesh\miniconda3\envs\525\lib\site-packages (from matplotlib) (2.8.1)
Requirement already satisfied: numpy>=1.16 in c:\users\vignesh\miniconda3\envs\525\lib\site-packages (from matplotlib) (1.20.2)
Requirement already satisfied: pyparsing>=2.2.1 in c:\users\vignesh\miniconda3\envs\525\lib\site-packages (from matplotlib) (2.4.7)
Requirement already satisfied: six in c:\users\vignesh\miniconda3\envs\525\lib\site-packages (from cycler[...]<jupyter_text>Loading in Chunks<jupyter_code>def get_counts(column, file = "../data/ACCESS-CM2_daily_rainfall_NSW.csv"):
    counts = pd.Series(dtype=int)

    for chunk in pd.read_csv(file, chunksize=10_000):
        counts = counts.add(chunk[column].value_counts(), fill_value=0)

    return counts
chunk_mem = %memit -o get_counts('lat_max')
chunk_time = %timeit -o get_counts('lat_max')<jupyter_output>peak memory: 217.21 MiB, increment: 12.07 MiB
5.03 s ± 710 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)<jupyter_text>Dask<jupyter_code>dask_mem = %memit -o dd.read_csv("../data/ACCESS-CM2_daily_rainfall_NSW.csv")['lat_max'].value_counts().compute()
dask_time = %timeit -o dd.read_csv("../data/ACCESS-CM2_daily_rainfall_NSW.csv")['lat_max'].value_counts().compute()<jupyter_output>peak memory: 628.03 MiB, increment: 421.87 MiB
1.68 s ± 20.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)<jupyter_text>Loading only columns of interest<jupyter_code>col_subset_mem = %memit -o pd.read_csv("../data/ACCESS-CM2_daily_rainfall_NSW.csv", usecols=['lat_max'])['lat_max'].value_counts()
col_subset_time = %timeit -o pd.read_csv("../data/ACCESS-CM2_daily_rainfall_NSW.csv", usecols=['lat_max'])['lat_max'].value_counts()<jupyter_output>peak memory: 245.17 MiB, increment: 39.38 MiB
1.21 s ± 37.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)<jupyter_text>Loading with `low_memory=True`<jupyter_code>low_mem = %memit -o pd.read_csv("../data/ACCESS-CM2_daily_rainfall_NSW.csv", usecols=['lat_max'],low_memory=True)['lat_max'].value_counts()
low_time = %timeit -o pd.read_csv("../data/ACCESS-CM2_daily_rainfall_NSW.csv", usecols=['lat_max'],low_memory=True)['lat_max'].value_counts()<jupyter_output>peak memory: 242.94 MiB, increment: 36.08 MiB
1.17 s ± 31.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)<jupyter_text>Comparison<jupyter_code>analysis = pd.DataFrame({
    "Method": ["Chunking", "Dask", "Subsetting Columns", "Low Memory"],
    "Time": [np.mean(chunk_time.all_runs), np.mean(dask_time.all_runs), np.mean(col_subset_time.all_runs), np.mean(low_time.all_runs)],
    "Memory": [chunk_mem.mem_usage[0], dask_mem.mem_usage[0], col_subset_mem.mem_usage[0], low_mem.mem_usage[0]]
})

analysis
fig, ax = plt.subplots(figsize=(8,6))
sc = ax.scatter(analysis.Time, analysis.Memory, c = pd.Categorical(analysis.Method).codes, cmap='Dark2')
ax.legend(sc.legend_elements()[0], analysis.Method, title="Method")
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15288153681963715, 'fraction_numerical': 0.04348986125933831, 'mean_word_length': 5.981378026070764, 'pattern_counts': {'":': 3, '<': 20, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '543741', 'license': 'MIT', 'n_tokens_mistral': 1559, 'n_tokens_neox': 1461, 'n_words': 276, 'path': 'notebooks/memory.ipynb', 'repo_name': 'UBC-MDS/figshare-group-5', 'size': '3748'}