<jupyter_start><jupyter_text>Compressing sparse datasetsValues in real-world data are often not evenly distributed.When there is a predominant value in your data, such as 0 or "missing", it's often more memory efficient to store only the different values. In this notebook we will let compressio consider sparse data structures for compression.You can find more information on how the SparseArray in pandas works on [this page](https://pandas.pydata.org/pandas-docs/stable/user_guide/sparse.html).<jupyter_code>%load_ext autoreload
%autoreload 2
import pandas as pd
import numpy as np

from compressio import Compress, storage_size, SparseCompressor, compress_report, savings_report<jupyter_output><empty_output><jupyter_text>We create a fictional cars dataset to illustrate sparse data structures can reduce memory usage with pandas. The amount of missing data is exaggerated for demonstation purposes: 95% of the data is missing for the sparse columns. You can play around with the parameters below to see for which amounts of missing values the sparse representation is more memory efficient.<jupyter_code>import random

n_missing = 19000
n_present = 1000
n_total = n_missing + n_present

data = pd.DataFrame({
    'primary_color': pd.Series(random.choices(['green', 'blue', 'red', 'yellow', 'white', 'pink'], k=n_total), dtype=str),
    'secondary_color': pd.Series([None] * n_missing + random.choices(['gold', 'black', 'silver'], k=n_present), dtype=str),
    'date_registered': pd.Series([1] * n_total, dtype="datetime64[ns]"),
    'date_scrapped': pd.Series([pd.NaT] * n_missing + random.choices([1,2,3,4,5,6,7,8,9,10],k=n_present), dtype="datetime64[ns]"),
    'number_of_modifications': pd.Series([pd.NA] * n_missing + random.choices([0,1,2,3,4],k=n_present), dtype="Int64"),
    'imported': pd.Series([pd.NA] * n_missing + random.choices([True, False], k=n_present), dtype="boolean"),
})

data
compress = Compress(compressor=SparseCompressor())
original_size = storage_size(data).to('megabyte')
print(f'Original DataFrame size: {original_size}')
compress.typeset.detect_type(data)
data_compressed = compress.it(data)
data_compressed
compress_report(data, compress.typeset, compress.compressor, with_inference=False, units="kilobytes")
savings_report(data, data_compressed)
data_compressed["secondary_color"].sparse.density
data_compressed["number_of_modifications"].sparse.density
data_compressed["imported"].sparse.density<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11693057247259439, 'fraction_numerical': 0.014616321559074299, 'mean_word_length': 7.158940397350993, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '176157', 'license': 'MIT', 'n_tokens_mistral': 786, 'n_tokens_neox': 748, 'n_words': 211, 'path': 'examples/notebooks/Sparse Data.ipynb', 'repo_name': 'dylan-profiler/compressio', 'size': '2463'}