<jupyter_start><jupyter_text>POC of reading/writing a feather dataframeThere is a nice writeup of file format comparisons [here](https://towardsdatascience.com/the-best-format-to-save-pandas-data-414dca023e0d), this notebook is just for my own exploration.With* 1M Rows* 10 Categorical Columns* 10 Numeric ColumnsWith Test 1 using `object` for the categorical columns| File Type | Write Speed | Read Speed | File Size ||-----------|-------------|------------|-----------|| Feather   | 954 ms      | 489 ms     | 443M      || CSV/GZ    | 34 s        | 8.47 s     | 143M      | With Test 2 using `pd.Categorical` for the categorical columns.  This is there Feather really shins as the categorical variables are heavily leveraged for reading/writing and file size.  Also when the dataframe is read in the categorical dtypes are automatically pulled from the file.  The csv file has object column types.| File Type | Write Speed | Read Speed | File Size ||-----------|-------------|------------|-----------|| Feather   | 60.9 ms     | 61.8 ms    | 88M       || CSV/GZ    | 32.8 s      | 7.99 s     | 143M      | (Note: These numbers may not 100% match below here, they are from a single run)<jupyter_code>%%capture
!conda install --yes -c conda-forge feather-format
!pip install hurry.filesize
# Number of numerical columns to be generated in the test dataframe
numerical_col_count = 10

# Number of categorical columns to be generated in the test dataframe
categorical_col_count = 10

# Number of rows for the test dataframe
n_rows = 1000000
import numpy as np
import pandas as pd
from pandas.testing import assert_frame_equal
import feather # Don't need this, just double checking that it's installed
import tempfile
import os
from hurry.filesize import size
# copied from https://towardsdatascience.com/the-best-format-to-save-pandas-data-414dca023e0d

def generate_dataset(n_rows, num_count, cat_count, max_nan=0.1, max_cat_size=100):
    """Randomly generate datasets with numerical and categorical features.
    
    The numerical features are taken from the normal distribution X ~ N(0, 1).
    The categorical features are generated as random uuid4 strings with 
    cardinality C where 2 <= C <= max_cat_size.
    
    Also, a max_nan proportion of both numerical and categorical features is replaces
    with NaN values.
    """
    dataset, types = {}, {}
    
    def generate_categories():
        from uuid import uuid4
        category_size = np.random.randint(2, max_cat_size)
        return [str(uuid4()) for _ in range(category_size)]
    
    for col in range(num_count):
        name = f'n{col}'
        values = np.random.normal(0, 1, n_rows)
        nan_cnt = np.random.randint(1, int(max_nan*n_rows))
        index = np.random.choice(n_rows, nan_cnt, replace=False)
        values[index] = np.nan
        dataset[name] = values
        types[name] = 'float32'
        
    for col in range(cat_count):
        name = f'c{col}'
        cats = generate_categories()
        values = np.array(np.random.choice(cats, n_rows, replace=True), dtype=object)
        nan_cnt = np.random.randint(1, int(max_nan*n_rows))
        index = np.random.choice(n_rows, nan_cnt, replace=False)
        values[index] = np.nan
        dataset[name] = values
        types[name] = 'object'
    
    return pd.DataFrame(dataset), types
df, types = generate_dataset(n_rows, numerical_col_count, categorical_col_count)
display(df.head(3))
display(types)<jupyter_output><empty_output><jupyter_text>Speeds for reading/writing* feather* gzip/csv<jupyter_code>out_feather_tmp_file = tempfile.NamedTemporaryFile(suffix='.feather')
out_gzip_csv_tmp_file = tempfile.NamedTemporaryFile(suffix='.csv.gz')<jupyter_output><empty_output><jupyter_text>Speed to write a feather file<jupyter_code>%%time
df.to_feather(out_feather_tmp_file.name)<jupyter_output>CPU times: user 683 ms, sys: 352 ms, total: 1.04 s
Wall time: 1.01 s<jupyter_text>Speed to read a feather file<jupyter_code>%%time
feather_df = pd.read_feather(out_feather_tmp_file.name)
assert_frame_equal(df, feather_df)<jupyter_output><empty_output><jupyter_text>Speed to write a gziped csv<jupyter_code>%%time
df.to_csv(out_gzip_csv_tmp_file.name, compression='gzip')<jupyter_output>CPU times: user 35.3 s, sys: 147 ms, total: 35.5 s
Wall time: 35.5 s<jupyter_text>Speed to read a gziped csv<jupyter_code>%%time
csv_gzip_df = pd.read_csv(out_gzip_csv_tmp_file.name)
try:
    assert_frame_equal(df, csv_gzip_df)
    print("SUCCESS: Frames are equal!")
except:
    print("FAILURE: Frames are unequal!")

# By default the csv file has an index read in as an additional column
assert_frame_equal(df, csv_gzip_df.drop(columns=['Unnamed: 0']))<jupyter_output>FAILURE: Frames are unequal!<jupyter_text>Comparison of File Sizes<jupyter_code>feather_size_mb = size(os.path.getsize(out_feather_tmp_file.name))
csv_gzip_size_mb = size(os.path.getsize(out_gzip_csv_tmp_file.name))

print(f"Size of Feather File in MB: {feather_size_mb}")
print(f"Size of csv/gzip File in MB: {csv_gzip_size_mb}")<jupyter_output>Size of Feather File in MB: 444M
Size of csv/gzip File in MB: 174M<jupyter_text>Use pd.CategoricalOne benefit of Feather is that it stores the type of the object.So let's try that with a categorical column<jupyter_code>for col in df.dtypes[df.dtypes == 'object'].index:
    df[col] = df[col].astype('category')
%%time
df.to_feather(out_feather_tmp_file.name)<jupyter_output>CPU times: user 136 ms, sys: 47.3 ms, total: 184 ms
Wall time: 62.8 ms<jupyter_text>Speed to read a feather file<jupyter_code>%%time
feather_df = pd.read_feather(out_feather_tmp_file.name)
assert_frame_equal(df, feather_df)<jupyter_output><empty_output><jupyter_text>Speed to write a gziped csv<jupyter_code>%%time
df.to_csv(out_gzip_csv_tmp_file.name, compression='gzip')<jupyter_output>CPU times: user 34.8 s, sys: 176 ms, total: 35 s
Wall time: 35 s<jupyter_text>Speed to read a gziped csv<jupyter_code>%%time
csv_gzip_df = pd.read_csv(out_gzip_csv_tmp_file.name)
try:
    # By default the csv file has an index read in as an additional column
    assert_frame_equal(df, csv_gzip_df.drop(columns=['Unnamed: 0']))
    print("SUCCESS: Frames are equal!")
except:
    print("FAILURE: Frames are unequal!")
    
print("Starting to see the benefits of Feather here as the dtypes differ")
display(pd.concat([df.dtypes, csv_gzip_df.dtypes], axis=1, sort=True)
        .rename(columns={0:'dtypes_from_original_df', 1:'dtypes_from_csv_gzip_df'}))<jupyter_output>FAILURE: Frames are unequal!
Starting to see the benefits of Feather here as the dtypes differ<jupyter_text>Comparison of File Sizes<jupyter_code>feather_size_mb = size(os.path.getsize(out_feather_tmp_file.name))
csv_gzip_size_mb = size(os.path.getsize(out_gzip_csv_tmp_file.name))

print(f"Size of Feather File in MB: {feather_size_mb}")
print(f"Size of csv/gzip File in MB: {csv_gzip_size_mb}")<jupyter_output>Size of Feather File in MB: 88M
Size of csv/gzip File in MB: 174M
{'fraction_non_alphanumeric': 0.10995970063327576, 'fraction_numerical': 0.02158894645941278, 'mean_word_length': 4.361882716049383, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 41, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '205018', 'license': 'MIT', 'n_tokens_mistral': 2518, 'n_tokens_neox': 2345, 'n_words': 655, 'path': 'feather_pandas/feather_pandas_poc.ipynb', 'repo_name': 'pettinato/poc_work', 'size': '6948'}