<commit_before>#python-hashes

Interesting (non-cryptographic) hashes implemented in pure Python. Included so far:

 * Bloom filters
 * Charikar similarity hashes 
 * Nilsimsa signatures
 * geohashes

Each hash is implemented as its own type extended from the base class `hashtype`.

Official repository and latest version: https://github.com/sangelone/python-hashes

To install the latest version, you can either do `easy_install python-hashes` or
`pip install python-hashes`. You may need to use `sudo`, depending on your environment.

---

###simhash

Charikar similarity is most useful for creating 'fingerprints' of
documents or metadata so you can quickly find duplicates or cluster
items. It operates on lists of strings, treating each word as its
own token (order does not matter, as in the bag-of-words model).

Here is a quick example session showing off similarity hashes:

    >>> from hashes.simhash import simhash
    >>> hash1 = simhash('This is a test string one.')
    >>> hash2 = simhash('This is a test string TWO.')
    >>> hash1
    <simhash.simhash object at 0x7f1f93070c90>
    >>> print hash1, hash2
    10203485745788768176630988232 10749932022170787621889701832
    >>> hash1.hex()
    '0x20f82026a01daffae45cfdc8L'
    >>> hash1.similarity(hash2)
    0.875                   # % of differing bits
    >>> long(hash1) - long(hash2)
    -546446276382019445258713600L
    >>> hash1 < hash2       # Hashes of the same type can be compared
    True
    >>> a_list = [hash2, hash1, 4.2]
    >>> for item in a_list: print item
    10749932022170787621889701832
    10203485745788768176630988232
    4.2<commit_msg>Clarification on Hamming distance<commit_after>#python-hashes

Interesting (non-cryptographic) hashes implemented in pure Python. Included so far:

 * Bloom filters
 * Charikar similarity hashes 
 * Nilsimsa signatures
 * geohashes

Each hash is implemented as its own type extended from the base class `hashtype`.

Official repository and latest version: https://github.com/sangelone/python-hashes

To install the latest version, you can either do `easy_install python-hashes` or
`pip install python-hashes`. You may need to use `sudo`, depending on your environment.

---

###simhash

Charikar similarity is most useful for creating 'fingerprints' of
documents or metadata so you can quickly find duplicates or cluster
items. It operates on lists of strings, treating each word as its
own token (order does not matter, as in the bag-of-words model).

Here is a quick example session showing off similarity hashes:

    >>> from hashes.simhash import simhash
    >>> hash1 = simhash('This is a test string one.')
    >>> hash2 = simhash('This is a test string TWO.')
    >>> hash1
    <simhash.simhash object at 0x7f1f93070c90>
    >>> print hash1, hash2
    10203485745788768176630988232 10749932022170787621889701832
    >>> hash1.hex()
    '0x20f82026a01daffae45cfdc8L'
    >>> hash1.similarity(hash2)
    0.875                   # % of bits in common (calculated via hamming distance)
    >>> long(hash1) - long(hash2)
    -546446276382019445258713600L
    >>> hash1 < hash2       # Hashes of the same type can be compared
    True
    >>> a_list = [hash2, hash1, 4.2]
    >>> for item in a_list: print item
    10749932022170787621889701832
    10203485745788768176630988232
    4.2
{'fraction_non_alphanumeric': 0.07947019867549669, 'fraction_numerical': 0.11318482841661649, 'mean_word_length': 3.6345885634588564, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 71, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6729440', 'n_tokens_mistral': 1260, 'n_tokens_neox': 1021, 'n_words': 367}