<commit_before># godedupe

Godedupe is a tool for finding duplicated or similar files.

It aims to be faster and provide a more intelligent approach for finding redundant files than old programs like fdupes.

The version 1.0.0 is our first version. In this version the only supported OS will be GNU/Linux and a lot of features are mising
but (at least) it is able to find duplicate files like fdupes does.

With the release of version 2.0.0 its planned to be able to find files which are almost identical. Useful for example when one file
is a modern version of other file, in this case the BLAKE sum will not be the same, but our program should be able to identify it.

### Version naming

We use an X.Y.Z style for naming our version. X means a new version that have some important new feature, Y means some minor feature
and Z means a correction.


### How we find duplicated files

For now our approach is very dummy. We simply declare a map of `Duplicated` where `Duplicated` is an struct like
```go
type Duplicated struct {
	listDuplicated []File
}
```

Then we hash with the BLAKE algorithm the file that we want to test and check if this hash is already in the map, if it's true we append the File we just
tested to the `listDuplicated` and report that there is a duplicated, if it's false we create a new `Duplicated` and append the
 file we just checked to `listDuplicated`.


### Resources

The images used are from [icons8](https://icons8.com/)<commit_msg>added link to blake repository in the readme<commit_after># godedupe

Godedupe is a tool for finding duplicated or similar files.

It aims to be faster and provide a more intelligent approach for finding redundant files than old programs like fdupes.

The version 1.0.0 is our first version. In this version the only supported OS will be GNU/Linux and a lot of features are mising
but (at least) it is able to find duplicate files like fdupes does.

With the release of version 2.0.0 its planned to be able to find files which are almost identical. Useful for example when one file
is a modern version of other file, in this case the BLAKE sum will not be the same, but our program should be able to identify it.

### Version naming

We use an X.Y.Z style for naming our version. X means a new version that have some important new feature, Y means some minor feature
and Z means a correction.


### How we find duplicated files

For now our approach is very dummy. We simply declare a map of `Duplicated` where `Duplicated` is an struct like
```go
type Duplicated struct {
	listDuplicated []File
}
```

Then we hash with the [BLAKE](https://github.com/minio/blake2b-simd) algorithm the file that we want to test and check if this hash is already in the map, if it's true we append the File we just
tested to the `listDuplicated` and report that there is a duplicated, if it's false we create a new `Duplicated` and append the
 file we just checked to `listDuplicated`.


### Resources

The images used are from [icons8](https://icons8.com/)
{'fraction_non_alphanumeric': 0.04886968085106383, 'fraction_numerical': 0.005651595744680851, 'mean_word_length': 4.031772575250836, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6377218', 'n_tokens_mistral': 817, 'n_tokens_neox': 780, 'n_words': 508}