<commit_before>dfm
===

Duplicate File Manager

The purpose of dfm is to locate duplicate files through a recursive search.

You can create an instance of the DFM object with optionally secifing the
directory path and the file extensions.

    instance = DFM.new( path: './', filters: ["jpg","png"] )

If you are using a ruby version before 2 then this would be:

    instance = DFM.new( { :path => './', :filters => ["jpg","png"] } )

Then you may get a hash of the hex matches (indacating files with identical content) by

    instance.hex

And by duplicate faile name by

    instance.name

Either of these can be called with false if you want only single instances of files.

    instance.hex( false )
    instance.name( false )

---

Also included is a command line version which outputs nicely formatted JSON in case
you would like to use it with anything else.  Type `dfm -h` on the command line to get
a list of available options.  Running `dfm` by itself will recursively search the current
folder for all duplicates by both file name and hash indexes.<commit_msg>spelling<commit_after>dfm 
=== 
 
Duplicate File Manager 
 
The purpose of dfm is to locate duplicate files through a recursive search. 
 
You can create an instance of the DFM object with optionally specifying the 
directory path and the file extensions. 
 
    dfm = DFM.new( path: './', filters: ["jpg","png"] ) 
 
If you are using a ruby version before 2 then this would be: 
 
    dfm = DFM.new( { :path => './', :filters => ["jpg","png"] } ) 
 
Then you may get a hash of the hex matches (indicating files with identical content) by 
 
    dfm.hex 
 
And by duplicate file name by 
 
    dfm.name 
 
Either of these can be called with false if you want only single instances of files. 
 
    dfm.hex( false ) 
    dfm.name( false ) 
 
--- 
 
Also included is a command line version which outputs nicely formatted JSON in case 
you would like to use it with anything else.  Type `dfm -h` on the command line to get 
a list of available options.  Running `dfm` by itself will recursively search the current 
folder for all duplicates by both file name and hash indexes. 
 
{'fraction_non_alphanumeric': 0.06793857608189856, 'fraction_numerical': 0.0009306654257794323, 'mean_word_length': 3.265873015873016, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6281329', 'n_tokens_mistral': 625, 'n_tokens_neox': 567, 'n_words': 319}