<commit_before># Dedup

Dedup is a command-line tool which de-deduplicates all provided directories by merging all into the first. The merge operation simultaneously removes duplicates and renames files due to path collisions.

### Install

**Binaries**

See [Releases](https://github.com/jpillora/dedup/releases/latest)

**Source**

``` sh
$ go get -v github.com/jpillora/dedup
```

### Usage

```
$ dedup --help
```

<tmpl,code: dedup --help>
```

	Usage: dedup [options] <dir> [dir] [dir]
	
	Version: 0.0.0-src

	Dedup de-deduplicates all provided directories by merging
	all into the first. The merge operation simultaneously
	removes duplicates and renames files due to path collisions.
	
	Options:
	  --keep, keep duplicates (by default, duplicates are deleted)
	  -v, verbose logs (display each move and delete)
	  --version, display version
	  -h --help, this help text

	Notes:
	  * dedup considers two files duplicates if they have
	    matching sha1 sums
	  * dedup is not recursive (only works on files)
	  * dedup is a destructive operation (unless --keep)
	  * dedup on a single directory will only perform
	    deduplication, no moves
	  * dedup renames: when a file is unique, dedup will
	    attempt to move the file. if the path already
	    exists the incoming file will be suffixed with
	    the next number (for example, if 'foo.txt' exists,
	    the new file will be 'foo-2.txt')
	  * any error will cause dedup to exit

	Read more: https://github.com/jpillora/dedup

```
</tmpl>

#### MIT License

Copyright © 2015 <NAME> &lt;<EMAIL>&gt;

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
'Software'), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.<commit_msg>added example output to docs<commit_after># Dedup

Dedup is a command-line tool which de-deduplicates all provided directories by merging all into the first. The merge operation simultaneously removes duplicates and renames files due to path collisions.

### Install

**Binaries**

See [Releases](https://github.com/jpillora/dedup/releases/latest)

**Source**

``` sh
$ go get -v github.com/jpillora/dedup
```

### Usage

```
$ dedup --help
```

<tmpl,code: dedup --help>
```

	Usage: dedup [options] <dir> [dir] [dir]
	
	Version: 0.0.0-src

	Dedup de-deduplicates all provided directories by merging
	all into the first. The merge operation simultaneously
	removes duplicates and renames files due to path collisions.
	
	Options:
	  --keep, keep duplicates (by default, duplicates are deleted)
	  -v, verbose logs (display each move and delete)
	  --version, display version
	  -h --help, this help text

	Notes:
	  * dedup considers two files duplicates if they have
	    matching sha1 sums
	  * dedup is not recursive (only works on files)
	  * dedup is a destructive operation (unless --keep)
	  * dedup on a single directory will only perform
	    deduplication, no moves
	  * dedup renames: when a file is unique, dedup will
	    attempt to move the file. if the path already
	    exists the incoming file will be suffixed with
	    the next number (for example, if 'foo.txt' exists,
	    the new file will be 'foo-2.txt')
	  * any error will cause dedup to exit

	Read more: https://github.com/jpillora/dedup

```
</tmpl>

### Example

``` sh
$ cd example/
$ tree .
.
├── bar
│   ├── bar.txt
│   ├── foo-copy.txt
│   └── foo.txt
└── foo
    ├── bazz.txt
    └── foo.txt

2 directories, 5 files

$ dedup foo bar
Indexing 'foo' (#2 files)
Merging in 'bar' (#3 files)
Done (moves 1 deletes 2)

$ tree .
.
├── bar
└── foo
    ├── bar.txt
    ├── bazz.txt
    └── foo.txt

2 directories, 3 files
```

#### MIT License

Copyright © 2015 <NAME> &lt;<EMAIL>&gt;

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
'Software'), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
{'fraction_non_alphanumeric': 0.09063573883161512, 'fraction_numerical': 0.005584192439862543, 'mean_word_length': 3.6663326653306614, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5573884', 'n_tokens_mistral': 1488, 'n_tokens_neox': 1342, 'n_words': 633}