<commit_before>---
layout: post
title: shrynk - Using Machine Learning to learn how to Compress
subtitle: 30% less disk required compared to using the single best strategy
---

I made the package [shrynk](https://github.com/kootenpv/shrynk) for compression using machine learning! It helps you by choosing (and applying) the format to compress your dataframes, JSON, or actually, files in general.

Given example data, it was able to compress using 30% overall less disk space using a mixed strategy compared to the best single compression algorithm (meaning: choosing any other single compression algorithm will be even worse).

You can try it for yourself (by uploading a CSV file) at [https://shrynk.ai](https://shrynk.ai).

<a href="https://shrynk.ai"><center><img src="/img/shrynkpage.png" width="200px" style="border: 1px solid #ee6e73;" /></center></a>

Bonus: If the algorithm has it wrong, the features of the data (not the data itself) will be added to the python package on the next release!

Next, I will explain Compression, Machine Learning and the library I've built in Python. You can also jump to a section using the links below.

<ul style="float: right">
<li><small><a href="#compression">Compression</a></small></li>
<li><small><a href="#compressing-tabular-data">Compressing tabular data</a></small></li>
<li><small><a href="#tabular-data-in-python">Tabular data in Python</a></small></li>
<li><small><a href="#running-benchmarks">Running benchmarks</a></small></li>
<li><small><a href="#where-the-idea-came-from">Where the idea came from</a></small></li>
<li><small><a href="#machine-learning">Machine Learning</a></small></li>
<li><small><a href="#machine-learning-in-shrynk">Machine Learning in Shrynk</a></small></li>
<li><small><a href="#usage">Usage</a></small></li>
<li><small><a href="#data-comes-packaged">Data comes packaged</a></small></li>
<li><small><a href="#how-well-does-it-work-cross-validation">How well does it work? Cross-validation</a></small></li>
<li><small><a href="#conclusion">Conclusion</a></small></li>
<li><small><a href="#whats-next">What's next</a></small></li>
</ul>
<commit_msg>quick links styling<commit_after>---
layout: post
title: shrynk - Using Machine Learning to learn how to Compress
subtitle: 30% less disk required compared to using the single best strategy
---

I made the package [shrynk](https://github.com/kootenpv/shrynk) for compression using machine learning! It helps you by choosing (and applying) the format to compress your dataframes, JSON, or actually, files in general.

Given example data, it was able to compress using 30% overall less disk space using a mixed strategy compared to the best single compression algorithm (meaning: choosing any other single compression algorithm will be even worse).

You can try it for yourself (by uploading a CSV file) at [https://shrynk.ai](https://shrynk.ai).

<a href="https://shrynk.ai"><center><img src="/img/shrynkpage.png" width="200px" style="border: 1px solid #ee6e73;" /></center></a>

Bonus: If the algorithm has it wrong, the features of the data (not the data itself) will be added to the python package on the next release!

Next, I will explain Compression, Machine Learning and the library I've built in Python. You can also jump to a section using the links below.

<div style="float: right; list-style-type: none;"><span style="padding-left: 40px">Quick links</span>
<ul>
<li><small><a href="#compression">Compression</a></small></li>
<li><small><a href="#compressing-tabular-data">Compressing tabular data</a></small></li>
<li><small><a href="#tabular-data-in-python">Tabular data in Python</a></small></li>
<li><small><a href="#running-benchmarks">Running benchmarks</a></small></li>
<li><small><a href="#where-the-idea-came-from">Where the idea came from</a></small></li>
<li><small><a href="#machine-learning">Machine Learning</a></small></li>
<li><small><a href="#machine-learning-in-shrynk">Machine Learning in Shrynk</a></small></li>
<li><small><a href="#usage">Usage</a></small></li>
<li><small><a href="#data-comes-packaged">Data comes packaged</a></small></li>
<li><small><a href="#how-well-does-it-work-cross-validation">How well does it work? Cross-validation</a></small></li>
<li><small><a href="#conclusion">Conclusion</a></small></li>
<li><small><a href="#whats-next">What's next</a></small></li>
</ul>
</div>

{'fraction_non_alphanumeric': 0.16980266177145478, 'fraction_numerical': 0.005507113354749885, 'mean_word_length': 5.7372488408037094, 'pattern_counts': {'":': 0, '<': 165, '<?xml version=': 0, '>': 165, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6722730', 'n_tokens_mistral': 1354, 'n_tokens_neox': 1295, 'n_words': 437}