<jupyter_start><jupyter_text>test across all columns (when is it worth running in this system)shrinking encodings of different type (approximate queries)ca_police 6.7GBfull_files 802MBcol_files 802 MBparse time 3min 36sPREDS[5:6] 22161713read_fast 8.66 sread 1min 24sread_chunks 2min 20sread_csv 2min 55sPREDS[4:10] 528503read_fast  6.58sread 2 min 7sread_chunks 2min 12sread_csv 3min 21sPREDS[0:20] 1read_fast 2.66 sread 6min 38sread_chunks 1min 59sread_csv 2min 58sgithub_issues 1min 27sPREDS[0:3] 1read_fast 51.9 sread 2min 56sread_chunks 49.9 sread_csv 1min 9sPREDS[0:1] 102077read_fast 2min 3sread 4min 43sread_chunks 1min 23sread_csv 2min 26sblock_total 22.6 sPREDS [2:3] 183368read_fast 795msread 29.5 sread_chunks 21.8 sread_csv 17.5 sPREDS [0:3] 1read_fast 1.92 sread 1min 13sread_chunks 26.3 sread_csv 29.6 sQueriesca_police[8] 82260698cardinality_one 234 mscardinality_many 447 msnormal 2-3 minca_police[8:10]cardinality_many 4.1sread_fast ~10 snormal ~2 minca_police[8:11]cardinality_many 6.57sread_fast ~10 snormal ~2 minca_police[8:13]cardinality_many 12.1sread_fast ~10 snormal ~2 min<jupyter_code>indices = []
    
df_all = []
for i in range(0,num_chunks+1):
    df_all.append(feather.read_dataframe(f'{FEATHER_DIR}full{i}.f'))
df = pd.concat(df_all, ignore_index=True)
print('concatted')
print(df.index)<jupyter_output>concatted
RangeIndex(start=0, stop=4999999, step=1)
{'fraction_non_alphanumeric': 0.08002883922134102, 'fraction_numerical': 0.14203316510454217, 'mean_word_length': 7.36144578313253, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '667682', 'license': 'PSF-2.0', 'n_tokens_mistral': 709, 'n_tokens_neox': 574, 'n_words': 129, 'path': 'Column_Storage.ipynb', 'repo_name': 'arjunrawal4/pandas-memdb', 'size': '1387'}