<commit_before># T-Digest

A map-reduce and parallel streaming friendly data-structure for accurate
quantile approximation.

This package provides a very crude implementation of Ted Dunning's t-digest
data structure in Go.

## Installation

    go get github.com/caio/go-tdigest

## Usage

    import ("github.com/caio/go-tdigest" "math/rand" "fmt")

    compression := 10
    t := tdigest.New(compression)

    for i := 0; i < 10000; i++ {
        t.Update(rand.Float64(), 1)
    }

    fmt.Printf("p(.5) = %.6f\n", t.Percentile(0.5))

## Disclaimer

I've written this solelly with the purpose of understanding how the
datastructure works, it hasn't been throughly verified nor have I bothered with
optimizations for now.

## References

This is a very simple port of the [reference][1] implementation with some
ideas borrowed from the [python version][2]. If you wanna get a quick grasp of
how it works and why it's useful, [this video and companion article is pretty
helpful].

[1]: https://github.com/tdunning/t-digest
[2]: https://github.com/CamDavidsonPilon/tdigest
[3]: https://www.mapr.com/blog/better-anomaly-detection-t-digest-whiteboard-walkthrough
<commit_msg>README fixup<commit_after># T-Digest

A map-reduce and parallel streaming friendly data-structure for accurate
quantile approximation.

This package provides a very crude implementation of Ted Dunning's t-digest
data structure in Go.

## Installation

    go get github.com/caio/go-tdigest

## Usage

    import ("github.com/caio/go-tdigest" "math/rand" "fmt")

    compression := 10
    t := tdigest.New(compression)

    for i := 0; i < 10000; i++ {
        t.Update(rand.Float64(), 1)
    }

    fmt.Printf("p(.5) = %.6f\n", t.Percentile(0.5))

## Disclaimer

I've written this solely with the purpose of understanding how the
data-structure works, it hasn't been throughly verified nor have I bothered with
optimizations for now.

## References

This is a very simple port of the [reference][1] implementation with some
ideas borrowed from the [python version][2]. If you wanna get a quick grasp of
how it works and why it's useful, [this video and companion article is pretty
helpful][3].

[1]: https://github.com/tdunning/t-digest
[2]: https://github.com/CamDavidsonPilon/tdigest
[3]: https://www.mapr.com/blog/better-anomaly-detection-t-digest-whiteboard-walkthrough

{'fraction_non_alphanumeric': 0.11125319693094629, 'fraction_numerical': 0.017476555839727195, 'mean_word_length': 3.8997912317327765, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 3, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6752717', 'n_tokens_mistral': 780, 'n_tokens_neox': 719, 'n_words': 257}