<commit_before># hllpp

[![Build Status](https://travis-ci.org/retailnext/hllpp.svg)](https://travis-ci.org/retailnext/hllpp) [![GoDoc](https://godoc.org/github.com/retailnext/hllpp?status.svg)](https://godoc.org/github.com/retailnext/hllpp)

hllpp is an implementation of the HyperLogLog++ cardinality estimation algorithm in go. It optimizes for memory usage over CPU usage. It implements all the HyperLogLog optimizations introduced in the HyperLogLog++ paper (http://goo.gl/Z5Sqgu). Some notable features include:

* marshaling so you can serialize to your datastore
* extra space savings by only using 5 bits per register when possible

## Usage

    h := hllpp.New()

    h.Add([]byte("barclay"))
    h.Add([]byte("reginald"))
    h.Add([]byte("barclay"))
    h.Add([]byte("broccoli"))<commit_msg>Add note about murmur3 to README.<commit_after># hllpp

[![Build Status](https://travis-ci.org/retailnext/hllpp.svg)](https://travis-ci.org/retailnext/hllpp) [![GoDoc](https://godoc.org/github.com/retailnext/hllpp?status.svg)](https://godoc.org/github.com/retailnext/hllpp)

hllpp is an implementation of the HyperLogLog++ cardinality estimation algorithm in go. It optimizes for memory usage over CPU usage. It implements all the HyperLogLog optimizations introduced in the HyperLogLog++ paper (http://goo.gl/Z5Sqgu). Some notable features include:

* marshaling so you can serialize to your datastore
* extra space savings by only using 5 bits per register when possible
* built-in non-streaming murmur3 implementation for fast hashing of input data

## Usage

    h := hllpp.New()

    h.Add([]byte("barclay"))
    h.Add([]byte("reginald"))
    h.Add([]byte("barclay"))
    h.Add([]byte("broccoli"))
{'fraction_non_alphanumeric': 0.1372434017595308, 'fraction_numerical': 0.0035190615835777126, 'mean_word_length': 4.802721088435374, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4274954', 'n_tokens_mistral': 571, 'n_tokens_neox': 548, 'n_words': 150}