<commit_before># Perceptive

Perceptive is a Go library which implements perceptual hash algorithms for 
comparing images.

Perceptual hash algorithms are a family of comparable hash functions which
generate distinct (but not unique) fingerprints, these fingerprints are then
comparable.

Perceptual hash algorithms are mainly used for detecting duplicates of the same 
files, in a way that standard and cryptographic hashes generally fail.

**Note:** This library can only compute hashes for images, it does not work on 
audio or video files.

Currently, the following perceptual hash algorithms are implemented:

- **Average Hash (Ahash)** - Fast but generates a huge number of false positives.
- **Difference Hash (Dhash)** - Fast and very few false positives.

**Perceptual hash algorithms can give false positives**, but there main aim is to 
give you a sense of similarity between files.

Perceptual hash algorithms tend to return a distance score. When comparing the 
two identical images below, we would receive a distance of 0:

<img src="https://github.com/umahmoood/perceptive/blob/master/perceptive_test/test_images/rainbow_flowers.jpg" with="310" height="300"/>
<img src="https://github.com/umahmoood/perceptive/blob/master/perceptive_test/test_images/rainbow_flowers.jpg" with="310" height="300"/>

A distance of zero means that the images are **likely** the same.

When comparing the two similar images below we would receive a distance between 
1-10 (depending on the hashing technique used):

<img src="https://github.com/umahmoood/perceptive/blob/master/perceptive_test/test_images/toy_story_1.jpg" width="440" height="290"/>
<img src="https://github.com/umahmoood/perceptive/blob/master/perceptive_test/test_images/toy_story_2.jpg" width="440" height="290"/>

A distance between 1-10 indicates the images are **likely a variation** of each 
other.

When comparing the two different images below we would receive a distance greater 
than 10:

<img src="https://github.com/umahmoood/perceptive/blob/master/perceptive_test/test_images/homer_doh.jpg" width="320" height="240"/>
<img src="https://github.com/umahmoood/perceptive/blob/master/perceptive_test/test_images/lena.jpg" width="320" height="240"/>
<commit_msg>fixed image urls as they were not loading<commit_after># Perceptive

Perceptive is a Go library which implements perceptual hash algorithms for 
comparing images.

Perceptual hash algorithms are a family of comparable hash functions which
generate distinct (but not unique) fingerprints, these fingerprints are then
comparable.

Perceptual hash algorithms are mainly used for detecting duplicates of the same 
files, in a way that standard and cryptographic hashes generally fail.

**Note:** This library can only compute hashes for images, it does not work on 
audio or video files.

Currently, the following perceptual hash algorithms are implemented:

- **Average Hash (Ahash)** - Fast but generates a huge number of false positives.
- **Difference Hash (Dhash)** - Fast and very few false positives.

**Perceptual hash algorithms can give false positives**, but there main aim is to 
give you a sense of similarity between files.

Perceptual hash algorithms tend to return a distance score. When comparing the 
two identical images below, we would receive a distance of 0:

<img src="https://github.com/umahmood/perceptive/blob/master/perceptive_test/test_images/rainbow_flowers.jpg" with="310" height="300"/>
<img src="https://github.com/umahmood/perceptive/blob/master/perceptive_test/test_images/rainbow_flowers.jpg" with="310" height="300"/>

A distance of zero means that the images are **likely** the same.

When comparing the two similar images below we would receive a distance between 
1-10 (depending on the hashing technique used):

<img src="https://github.com/umahmood/perceptive/blob/master/perceptive_test/test_images/toy_story_1.jpg" width="440" height="290"/>
<img src="https://github.com/umahmood/perceptive/blob/master/perceptive_test/test_images/toy_story_2.jpg" width="440" height="290"/>

A distance between 1-10 indicates the images are **likely a variation** of each 
other.

When comparing the two different images below we would receive a distance greater 
than 10:

<img src="https://github.com/umahmood/perceptive/blob/master/perceptive_test/test_images/homer_doh.jpg" width="320" height="240"/>
<img src="https://github.com/umahmood/perceptive/blob/master/perceptive_test/test_images/lena.jpg" width="320" height="240"/>

{'fraction_non_alphanumeric': 0.0922871154703522, 'fraction_numerical': 0.020954079358002677, 'mean_word_length': 5.319718309859155, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 12, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4343433', 'n_tokens_mistral': 1369, 'n_tokens_neox': 1236, 'n_words': 479}