<commit_before>simdcomp
========

A simple C library for compressing lists of integers using binary packing and SIMD instructions.

What is it for?
-------------

This is a low-level library for fast integer compression. By design it does not define a compressed
format. It is up to the (sophisticated) user to create a compressed format.

Requirements
-------------

- Your processor should support SSE2 (Pentium4 or better)
- C99 compliant compiler (GCC is assumed)
- A Linux-like distribution is assumed by the makefile

Usage
-------

Compression works over blocks of 128 integers.

For a complete working example, see example.c (you can build it and
run it with "make example; ./example").



1) Lists of integers in random order.
<commit_msg>Saving comment<commit_after>simdcomp
========

A simple C library for compressing lists of integers using binary packing and SIMD instructions.

This library can decode billions of compressed integers per second on most
desktop or laptop processors.

What is it for?
-------------

This is a low-level library for fast integer compression. By design it does not define a compressed
format. It is up to the (sophisticated) user to create a compressed format.

Requirements
-------------

- Your processor should support SSE2 (Pentium4 or better)
- C99 compliant compiler (GCC is assumed)
- A Linux-like distribution is assumed by the makefile

Usage
-------

Compression works over blocks of 128 integers.

For a complete working example, see example.c (you can build it and
run it with "make example; ./example").



1) Lists of integers in random order.

{'fraction_non_alphanumeric': 0.09170305676855896, 'fraction_numerical': 0.009981285090455396, 'mean_word_length': 3.4065934065934065, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7402745', 'n_tokens_mistral': 438, 'n_tokens_neox': 404, 'n_words': 229}