<commit_before>Usage
=====
hash-directory.py <directory whoms content will be walked and hashed> <where to create directory that will hold hashes>

Example
=====
./hash-directory.py /tmp /home/tmp

It will enumerate all the files, recursively, in /tmp and replicate the directory hierarchy in /home/tmp and place a file with the name _<filename> at the same place in the hierarchy as it finds it, but the content will be the has of the first 4MB of the file<commit_msg>Added backticks around the usage and filename to get it none escaped<commit_after>Usage
=====
`hash-directory.py <directory whoms content will be walked and hashed> <where to create directory that will hold hashes>`

Example
=====
./hash-directory.py /tmp /home/tmp

It will enumerate all the files, recursively, in /tmp and replicate the directory hierarchy in /home/tmp and place a file with the name `_<filename>` at the same place in the hierarchy as it finds it, but the content will be the has of the first 4MB of the file
{'fraction_non_alphanumeric': 0.07221664994984955, 'fraction_numerical': 0.0020060180541624875, 'mean_word_length': 4.423913043478261, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7069920', 'n_tokens_mistral': 270, 'n_tokens_neox': 257, 'n_words': 152}