<commit_before># nupes - WIP
## A command line tool for finding duplicate files
nupes helps you find duplicate files.

This is a work in progress.

* * *
### Installation
```bash
$ npm install -g nupes
```
* * *
### Usage
`nupes <directory>` will search that directory for duplicate files and output the result. `nupes` will do the same for the current directory.

* * *
### How does it work
nupes works by running a list of differentiation methods - each one seperates the current pools of files into smaller pools. Pools of size 1 are then ommited. The current differentiation methods are:
* File size
* Whole file md5

Generally, each method has a different trade-off of speed and accuracy. By starting with the cheapest methods, we make sure that more expensive ones will be executed on a smaller amount of files. By finishing with a whole-file-checksum, we guarantee that files deemed identical are indeed so.

* * *
### TODOs
* Add more differentiation methods: by headers, partial checksums.
* Optimise order of differentiation methods.
* Add option to limit files checked to certain sizes.
* Add option to sort results by containing folders, or by size, or by amount of duplicates.
* Add option to include or exclude files/folders by name or regex (currently excluding node_modules and .git).
* Add verbose and quite options and set output levels.
* Pretify output.
* Find duplicate folders.
* Turn this list into github issues.
* ...<commit_msg>added todo - sum of sizes.<commit_after># nupes - WIP
## A command line tool for finding duplicate files
nupes helps you find duplicate files.

This is a work in progress.

* * *
### Installation
```bash
$ npm install -g nupes
```
* * *
### Usage
`nupes <directory>` will search that directory for duplicate files and output the result. `nupes` will do the same for the current directory.

* * *
### How does it work
nupes works by running a list of differentiation methods - each one seperates the current pools of files into smaller pools. Pools of size 1 are then ommited. The current differentiation methods are:
* File size
* Whole file md5

Generally, each method has a different trade-off of speed and accuracy. By starting with the cheapest methods, we make sure that more expensive ones will be executed on a smaller amount of files. By finishing with a whole-file-checksum, we guarantee that files deemed identical are indeed so.

* * *
### TODOs
* Add more differentiation methods: by headers, partial checksums.
* Optimise order of differentiation methods.
* Add option to limit files checked to certain sizes.
* Add option to sort results by containing folders, or by size, or by amount of duplicates.
* Add option to include or exclude files/folders by name or regex (currently excluding node_modules and .git).
* Add verbose and quite options and set output levels.
* Add sum of all duplicate files sizes.
* Pretify output.
* Find duplicate folders.
* Turn this list into github issues.
* ...
{'fraction_non_alphanumeric': 0.06484295845997974, 'fraction_numerical': 0.0013508949679162446, 'mean_word_length': 4.098106712564544, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '275361', 'n_tokens_mistral': 792, 'n_tokens_neox': 739, 'n_words': 441}