<commit_before># olio
## A general support/utility library in C.

Not all modules have been implemented.

The most interesting bits are probably the random number generator (Mersenne Twister with the revised seeding procedure) and the skiplist (using a paged, deterministic algorithm).

Actual documentation to come some day soon.<commit_msg>Forgot to mention some other interesting bits in the readme.<commit_after># olio

## A general support/utility library in C.

Not all modules have been implemented.

The most interesting bits are probably the random number generator (Mersenne Twister with the revised seeding procedure); the skiplist (using a paged, deterministic algorithm); the wavelet-based, k-means clustering code; and maybe the graph dfs and perfect hash generator.

Actual documentation to come some day soon.
{'fraction_non_alphanumeric': 0.04727272727272727, 'fraction_numerical': 0.0, 'mean_word_length': 5.118518518518519, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6378011', 'n_tokens_mistral': 206, 'n_tokens_neox': 197, 'n_words': 113}