<commit_before>Re-pair-gnu
===========

This is an implementation of the string re-pairing algorithm, intended for use in detecting text redundancy. Compiled with g++ on windows using mingw.

The input is several text files (the versions of a document). The output is a partitioning of each version such as many redundant fragments as possible show up in the partitions of each version. So for example, here are three versions of a (very) small document:

Version 1 -> Alice was beginning to get very tired of sitting by her sister on the bank, and of having nothing to do:

Version 2 -> Alice was starting to get very tired of sitting by her sister on the bank, and of having nothing to do:

Version 3 -> Alice was beginning to get very tired of sitting by her sister by the lake, and of having nothing to do:

Ignoring punctuation and spacing, we want to identify the common sequences of words, so first we convert the text to integer arrays:

Version 1 -> [1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 8 17 18 4 19]

Version 2 -> [1 2 20 4 5 6 7 8 9 10 11 12 13 14 15 16 8 17 18 4 19]

Version 3 -> [1 2 3 4 5 6 7 8 9 10 11 12 10 14 21 16 8 17 18 4 19]

A good paritioning will find repeating fragments across versions, so first we identify those fragments:
A -> [1 2 3 4 5 6 7 8 9 10 11 12]
B -> [13 14 15 16 8 17 18 4 19]
C -> [1 2 20 4 5 6 7 8 9 10 11 12]
D -> [10 14 21 16 8 17 18 4 19]

Applying this back to the versions gives us:

Version 1 -> [A, B]

Version 2 -> [C, B]

Version 3 -> [A, D]<commit_msg>fixed readme spacing<commit_after>Re-pair-gnu
===========

This is an implementation of the string re-pairing algorithm, intended for use in detecting text redundancy. Compiled with g++ on windows using mingw.

The input is several text files (the versions of a document). The output is a partitioning of each version such as many redundant fragments as possible show up in the partitions of each version. So for example, here are three versions of a (very) small document:

Version 1 -> Alice was beginning to get very tired of sitting by her sister on the bank, and of having nothing to do:

Version 2 -> Alice was starting to get very tired of sitting by her sister on the bank, and of having nothing to do:

Version 3 -> Alice was beginning to get very tired of sitting by her sister by the lake, and of having nothing to do:

Ignoring punctuation and spacing, we want to identify the common sequences of words, so first we convert the text to integer arrays:

Version 1 -> [1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 8 17 18 4 19]

Version 2 -> [1 2 20 4 5 6 7 8 9 10 11 12 13 14 15 16 8 17 18 4 19]

Version 3 -> [1 2 3 4 5 6 7 8 9 10 11 12 10 14 21 16 8 17 18 4 19]

A good paritioning will find repeating fragments across versions, so first we identify those fragments:

A -> [1 2 3 4 5 6 7 8 9 10 11 12]

B -> [13 14 15 16 8 17 18 4 19]

C -> [1 2 20 4 5 6 7 8 9 10 11 12]

D -> [10 14 21 16 8 17 18 4 19]

Applying this back to the versions gives us:

Version 1 -> [A, B]

Version 2 -> [C, B]

Version 3 -> [A, D]
{'fraction_non_alphanumeric': 0.06012553683515032, 'fraction_numerical': 0.1096795507102742, 'mean_word_length': 3.1881051175656987, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1234740', 'n_tokens_mistral': 1200, 'n_tokens_neox': 858, 'n_words': 623}