<commit_before>Kamban
======

Kamban is a simple tool for quantizing the creativity of given text content. This was written hurriedly in a couple of hours. It is a pretty naive tool.

### Notes

1. adjudge.py takes a english text file as input and calculates
counts needed to find the creativity index. it takes time as
it keeps downloading pages from wikipedia. redirect the output
of this code to a file.

    `python adjudge.py -i license_1.txt > result.txt`

2. quantize.py reads the result.txt from standard input and does the calculation and
prints the creativity index - last line of the output.

    `python quantize.py < result.txt`

3. to tweak the calculation, tweak the formula at the end of quantize.py

4. `license_1.txt` and `80days_3.txt` are sample test files and their respective
result files are names with the suffix results.txt

5. Thank you <NAME> (<EMAIL>) for the stemmer.py!

6. Tweet me @annacoder for feedback.<commit_msg>link to the blog entry about Kamban<commit_after>Kamban
======

Kamban is a simple tool for quantizing the creativity of given text content. This was written hurriedly in a couple of hours. It is a pretty naive tool.

### Notes

1. adjudge.py takes a english text file as input and calculates
counts needed to find the creativity index. it takes time as
it keeps downloading pages from wikipedia. redirect the output
of this code to a file.

    `python adjudge.py -i license_1.txt > result.txt`

2. quantize.py reads the result.txt from standard input and does the calculation and
prints the creativity index - last line of the output.

    `python quantize.py < result.txt`

3. to tweak the calculation, tweak the formula at the end of quantize.py

4. `license_1.txt` and `80days_3.txt` are sample test files and their respective
result files are names with the suffix results.txt

5. Thank you <NAME> (<EMAIL>) for the stemmer.py!

6. Tweet me @annacoder for feedback.

7. More details at http://venkat.io/posts/kamban-quantizing-creativity-of-written-content/
{'fraction_non_alphanumeric': 0.06610337972166998, 'fraction_numerical': 0.01143141153081511, 'mean_word_length': 3.87409200968523, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4241126', 'n_tokens_mistral': 590, 'n_tokens_neox': 565, 'n_words': 296}