<commit_before># filemerger
A filemerger is someone who will merge duplicated files on your disk. Basically it will:

1. calculate every file's md5 in your disk(not whole disk but specific directory recommanded)
2. store file name(absolute path) and md5 in a key-value (filename-md5) store
3. store md5 and count in a key-value(md5-count) store
4. if a md5 shows twice or more, get the filenames and make result
5. show results and let user decide which file will be removed
6. one of the duplicated file will be saved, and the others will be deleted and use soft link file instead.

challenges:

1. bulk files high performance
2. file security (how to choose which file will be saved ?)<commit_msg>add steps and updated challenges<commit_after># filemerger
A filemerger is someone who will merge duplicated files on your disk. Basically it will:

1. calculate every file's md5 in your disk(not whole disk but specific directory recommanded)
2. store file name(absolute path) and md5 in a key-value (filename-md5) store
3. store md5 and count in a key-value(md5-count) store
4. if a md5 shows twice or more, get the filenames and make result
5. show results and let user decide which file will be removed
6. one of the duplicated file will be saved, and the others will be deleted and use soft link file instead.

steps:

1. present working directory (only files)
2. present working directory (iterated directories)
3. specified working directory

challenges:

1. bulk files high performance (velocity, memory pressure, etc)
2. file security (how to choose which file will be saved ?)
{'fraction_non_alphanumeric': 0.050505050505050504, 'fraction_numerical': 0.019570707070707072, 'mean_word_length': 4.3367003367003365, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4371479', 'n_tokens_mistral': 435, 'n_tokens_neox': 419, 'n_words': 257}