<jupyter_start><jupyter_text>Optimising memory and disk usageStrategies have been taken from:https://www.kaggle.com/frankherfert/tips-tricks-for-working-with-large-datasets  Import Statements<jupyter_code>import pandas as pd
import os<jupyter_output><empty_output><jupyter_text>Classes and Methods<jupyter_code>class dataframe_optimiser():
    def __init__(self, conversion):
        self.conversion = conversion
        
    def convert_series(self,array_of_series):
        returned_series=[]
        for series in array_of_series:
            print("converting: ",series.name,"\t\t\tsize (MB):", round(series.memory_usage(deep=True) * 1e-6,2),end="\t")
#             series=eval("series"+test.conversion)
            series=eval(self.conversion)
            print("->\t", round(series.memory_usage(deep=True) * 1e-6,2))
            returned_series.append(series)
        return returned_series<jupyter_output><empty_output><jupyter_text>Load Raw Datahttps://stackoverflow.com/questions/14262433/large-data-work-flows-using-pandas/1426880414268804<jupyter_code>%%time
i=1
chunksize = 10 ** 6
for chunk in pd.read_csv("../data/raw/train.csv", chunksize=chunksize):
    i+=1
    print(i)
    print(chunk.head())
    if i==2:
        break
        
sample_data=chunk
sample_data.head()<jupyter_output><empty_output><jupyter_text>Investigating memory usage<jupyter_code>%%time
sample_data.info(memory_usage="deep")
%%time
sample_data.memory_usage(deep=True) * 1e-6
initial_memory_usage=sample_data.memory_usage(deep=True).sum() * 1e-6
print("The space on disk is (MB): ",round(initial_memory_usage,2))<jupyter_output>The space on disk is (MB):  770.12<jupyter_text>Timestamp conversionConverting the timestamp to a numpy datetime type reduces the memory usage by a factor of 10.<jupyter_code>#instantiate category_converter object from dataframe_optimiser class
category_converter=dataframe_optimiser("pd.to_datetime(series)")

#convert array of series from dataframe to optimised datatype
[sample_data["timestamp"]]=category_converter.convert_series([sample_data["timestamp"]])<jupyter_output>converting:  timestamp 			size (MB): 81.0	->	 8.0<jupyter_text>Categorical value conversionConverting the categorical values from strings to a numerical index leads to reductions of up to 75x<jupyter_code>#instantiate category_converter object from dataframe_optimiser class
category_converter=dataframe_optimiser("series.astype('category')")

#convert array of series from dataframe to optimised datatype
[sample_data["title"],sample_data["world"],sample_data["event_code"],sample_data["event_count"],sample_data["installation_id"]]=category_converter.convert_series([sample_data["title"],sample_data["world"],sample_data["event_code"],sample_data["event_count"],sample_data["installation_id"]])<jupyter_output>converting:  title 			size (MB): 75.01	->	 1.0
converting:  world 			size (MB): 67.37	->	 1.0
converting:  event_code 			size (MB): 8.0	->	 1.0
converting:  event_count 			size (MB): 8.0	->	 2.1
converting:  installation_id 			size (MB): 65.0	->	 2.19<jupyter_text>Final Data Frame<jupyter_code>sample_data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 1000000 entries, 0 to 999999
Data columns (total 11 columns):
event_id           1000000 non-null object
game_session       1000000 non-null object
timestamp          1000000 non-null datetime64[ns, UTC]
event_data         1000000 non-null object
installation_id    1000000 non-null category
event_count        1000000 non-null category
event_code         1000000 non-null category
game_time          1000000 non-null int64
title              1000000 non-null category
type               1000000 non-null object
world              1000000 non-null category
dtypes: category(5), datetime64[ns, UTC](1), int64(1), object(4)
memory usage: 52.6+ MB<jupyter_text>Final size in memoryThe size in memory has been reduced by almost a factor of 2.It will be determined whether the conversion to category impacts the machine learning models.<jupyter_code>final_memory_usage=sample_data.memory_usage(deep=True).sum() * 1e-6
print("The space on disk is (MB): ",round(initial_memory_usage,2))
print("The initial space on disk was (MB): ",round(initial_memory_usage,2)," and the final space on disk is (MB): ",round(final_memory_usage,2))<jupyter_output>The initial space on disk was (MB):  770.12  and the final space on disk is (MB):  481.03<jupyter_text>Store data as a pickleBy storing the data as a pickle, the size on disk is reduced to 67% of the original size, and loads 10x faster.<jupyter_code>sample_data.to_csv("../data/processed/sample_data.csv")
sample_data.to_pickle("../data/processed/sample_data.pkl")
# size is shown in bytes again and needs to be converted to megabytes
print("sample_data.csv (MB):", round(os.stat('../data/processed/sample_data.csv').st_size * 1e-6,2))
print("sample_data.pkl (MB):", round(os.stat('../data/processed/sample_data.pkl').st_size * 1e-6,2))
%%time
train = pd.read_pickle("../data/processed/sample_data.pkl")
%%time
train = pd.read_csv("../data/processed/sample_data.csv")<jupyter_output>CPU times: user 4.61 s, sys: 375 ms, total: 4.98 s
Wall time: 4.43 s
{'fraction_non_alphanumeric': 0.11673753382296095, 'fraction_numerical': 0.043873212214920757, 'mean_word_length': 5.1387900355871885, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 39, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '319248', 'license': 'FTL', 'n_tokens_mistral': 1861, 'n_tokens_neox': 1685, 'n_words': 391, 'path': 'notebooks/ls-0.1-initial-exploration-train-data.ipynb', 'repo_name': 'TheKaggleKings/kaggle-data-science-bowl', 'size': '5174'}