<commit_before># Corasick Park

Corasick Park is a server for quickly applying lots of transformations
to strings. By using a simple JSON interface, you can specify the
string replacements that should be made. After you describe the
transformations that should be made, you can hit another JSON endpoint
to quickly apply the transformations that you described.

Corasick Park is able to apply hundreds of thousands of transactions
in less than a second by using
[the Aho-Corasick algorithm](http://en.wikipedia.org/wiki/Aho%E2%80%93Corasick_string_matching_algorithm)
for finding out which transformations apply to the string you
provide. Currently the server uses the [AhoCorasick](Aho-Corasick
implementation in Haskell) for the heavy lifting of finding the
correct patterns to apply.

Since it is very quick even to precompile hundreds of thousands of
patterns, the server doesn't persist anything to disk, and uses
Haskell's MVars for safe concurrency.

Corasick Park allows you to precompile patterns in groups by a
'bucket' name so that you can easily choose which set of
transformations to apply for each string.<commit_msg>README change<commit_after># Corasick Park

Corasick Park is a server for quickly applying lots of transformations
to strings. By using a simple JSON interface, you can specify the
string replacements that should be made. After you describe the
transformations that should be made, you can hit another JSON endpoint
to quickly apply the transformations that you described.

Corasick Park is able to apply hundreds of thousands of transactions
in less than a second by using
[the Aho-Corasick algorithm](http://en.wikipedia.org/wiki/Aho%E2%80%93Corasick_string_matching_algorithm)
for finding out which transformations apply to the string you
provide. Currently the server uses the
[Aho-Corasick implementation in Haskell](http://hackage.haskell.org/package/AhoCorasick)
for the heavy lifting of finding the correct patterns to apply.

Since it is very quick even to precompile hundreds of thousands of
patterns, the server doesn't persist anything to disk, and uses
Haskell's MVars for safe concurrency.

Corasick Park allows you to precompile patterns in groups by a
'bucket' name so that you can easily choose which set of
transformations to apply for each string.
{'fraction_non_alphanumeric': 0.037004788855028295, 'fraction_numerical': 0.0043535045711798, 'mean_word_length': 5.244565217391305, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2667531', 'n_tokens_mistral': 591, 'n_tokens_neox': 557, 'n_words': 324}