<commit_before>[![Build Status](https://travis-ci.org/stripe/veneur.svg?branch=master)](https://travis-ci.org/stripe/veneur)
[![GoDoc](https://godoc.org/github.com/stripe/veneur?status.svg)](http://godoc.org/github.com/stripe/veneur)

Veneur (venn-urr) is a server implementation of the [DogStatsD protocol](http://docs.datadoghq.com/guides/dogstatsd/#datagram-format), which is a superset of the StatsD protocol.

# Status

Veneur is currently handling all metrics for Stripe and is considered production ready. It is, however, still in heavy development and may change!

Building veneur requires Go 1.7 or later.

# Motivation

We wanted percentiles, histograms and sets to be global. Veneur helps us do that!

Veneur is a DogStatsD implementation that acts as a local collector and — when configured differently — as an aggregator for
some metric types, such that the metrics are *global* rather than host-local. This is particularly useful for histograms,
timers and sets, as in their normal, per-host configuration the percentiles for histograms can be less effective or even
meaningless. Per-host unique sets are also often not what's desired.

Global \*StatsD installations can be problematic, as they either require client-side or proxy sharding behavior to prevent an
instance being a Single Point of Failure (SPoF) for all metrics. Veneur aims to solve this problem. Non-global
metrics like counters and gauges are collected by a local Veneur instance and sent to Datadog at flush time. Global metrics (histograms and sets)
are forwarded to a central Veneur instance for aggregation before being sent to Datadog.

# How Veneur Is Different Than Official DogStatsD

Veneur is different for a few reasons. They are enumerated here.

## Protocol

Veneur adheres to [the official DogStatsD datagram format](http://docs.datadoghq.com/guides/dogstatsd/#datagram-format) with the exceptions below:

* The tag `veneurlocalonly` is stripped and influenced forwarding behavior, as discussed below.

## Global Aggregation

If configured to do so, Veneur can selectively aggregate global metrics to be cumulative across all instances that report to a central Veneur, allowing global percentile
calculation and global set counts.

For example, say you emit a timer `foo.bar.call_duration_ms` from 20 hosts that are configured to forward to a central veneur. In Datadog you'll see the following:

* Metrics that have been "globalized"
  * `foo.bar.call_duration_ms.50percentile`: the p50 across all hosts, by tag
  * `foo.bar.call_duration_ms.90percentile`: the p90 across all hosts, by tag
  * `foo.bar.call_duration_ms.95percentile`: the p95 across all hosts, by tag
  * `foo.bar.call_duration_ms.99percentile`: the p99 across all hosts, by tag
* Metrics that remain host-local
  * `foo.bar.call_duration_ms.count`: by-host tagged count which (when summed) shows the total count of times this metric was emitted
  * `foo.bar.call_duration_ms.max`: by-host tagged maximum value
  * `foo.bar.call_duration_ms.min`: by-host tagged minimum value

Clients can choose to override this behavior by [including the tag `veneurlocalonly`](#magic-tag).

## Approximate Histograms

Because Veneur is built to handle lots and lots of data, it uses approximate histograms. We have our own implementation of [Dunning's t-digest](tdigest/merging_digest.go), which has bounded memory consumption and reduced error at extreme quantiles. Metrics are consistently routed to the same worker to distribute load and to be added to the same histogram.

Datadog's DogStatsD — and StatsD — uses an exact histogram which retains all samples and is reset every flush period. This means that there is a loss of precision when using Veneur, but
the resulting percentile values are meant to be more representative of a global view.

Veneur's timers and histograms do not emit an `avg` metric. Averages suck.

## Approximate Sets

Veneur uses [HyperLogLogs](https://github.com/clarkduvall/hyperloglog) for approximate unique sets. These are a very efficient unique counter with fixed memory consumption.

## Lack of Host Tags for Aggregated Metrics

By definition the hostname is not applicable to global metrics that Veneur processes. Note that if you
do include a hostname tag, Veneur will **not** strip it for you. Veneur will add it's own hostname as configured to metrics sent to Datadog.

## Expiration

Veneur expires all metrics on each flush. If a metric is no longer being sent (or is sent sparsely) Veneur will not send it as zeros! This was chosen because the combination of the approximation's features and the additional hysteresis imposed by *retaining* these approximations over time was deemed more complex than desirable.
<commit_msg>we do emit averages now<commit_after>[![Build Status](https://travis-ci.org/stripe/veneur.svg?branch=master)](https://travis-ci.org/stripe/veneur)
[![GoDoc](https://godoc.org/github.com/stripe/veneur?status.svg)](http://godoc.org/github.com/stripe/veneur)

Veneur (venn-urr) is a server implementation of the [DogStatsD protocol](http://docs.datadoghq.com/guides/dogstatsd/#datagram-format), which is a superset of the StatsD protocol.

# Status

Veneur is currently handling all metrics for Stripe and is considered production ready. It is, however, still in heavy development and may change!

Building veneur requires Go 1.7 or later.

# Motivation

We wanted percentiles, histograms and sets to be global. Veneur helps us do that!

Veneur is a DogStatsD implementation that acts as a local collector and — when configured differently — as an aggregator for
some metric types, such that the metrics are *global* rather than host-local. This is particularly useful for histograms,
timers and sets, as in their normal, per-host configuration the percentiles for histograms can be less effective or even
meaningless. Per-host unique sets are also often not what's desired.

Global \*StatsD installations can be problematic, as they either require client-side or proxy sharding behavior to prevent an
instance being a Single Point of Failure (SPoF) for all metrics. Veneur aims to solve this problem. Non-global
metrics like counters and gauges are collected by a local Veneur instance and sent to Datadog at flush time. Global metrics (histograms and sets)
are forwarded to a central Veneur instance for aggregation before being sent to Datadog.

# How Veneur Is Different Than Official DogStatsD

Veneur is different for a few reasons. They are enumerated here.

## Protocol

Veneur adheres to [the official DogStatsD datagram format](http://docs.datadoghq.com/guides/dogstatsd/#datagram-format) with the exceptions below:

* The tag `veneurlocalonly` is stripped and influenced forwarding behavior, as discussed below.

## Global Aggregation

If configured to do so, Veneur can selectively aggregate global metrics to be cumulative across all instances that report to a central Veneur, allowing global percentile
calculation and global set counts.

For example, say you emit a timer `foo.bar.call_duration_ms` from 20 hosts that are configured to forward to a central veneur. In Datadog you'll see the following:

* Metrics that have been "globalized"
  * `foo.bar.call_duration_ms.50percentile`: the p50 across all hosts, by tag
  * `foo.bar.call_duration_ms.90percentile`: the p90 across all hosts, by tag
  * `foo.bar.call_duration_ms.95percentile`: the p95 across all hosts, by tag
  * `foo.bar.call_duration_ms.99percentile`: the p99 across all hosts, by tag
* Metrics that remain host-local
  * `foo.bar.call_duration_ms.count`: by-host tagged count which (when summed) shows the total count of times this metric was emitted
  * `foo.bar.call_duration_ms.max`: by-host tagged maximum value
  * `foo.bar.call_duration_ms.min`: by-host tagged minimum value

Clients can choose to override this behavior by [including the tag `veneurlocalonly`](#magic-tag).

## Approximate Histograms

Because Veneur is built to handle lots and lots of data, it uses approximate histograms. We have our own implementation of [Dunning's t-digest](tdigest/merging_digest.go), which has bounded memory consumption and reduced error at extreme quantiles. Metrics are consistently routed to the same worker to distribute load and to be added to the same histogram.

Datadog's DogStatsD — and StatsD — uses an exact histogram which retains all samples and is reset every flush period. This means that there is a loss of precision when using Veneur, but
the resulting percentile values are meant to be more representative of a global view.

## Approximate Sets

Veneur uses [HyperLogLogs](https://github.com/clarkduvall/hyperloglog) for approximate unique sets. These are a very efficient unique counter with fixed memory consumption.

## Lack of Host Tags for Aggregated Metrics

By definition the hostname is not applicable to global metrics that Veneur processes. Note that if you
do include a hostname tag, Veneur will **not** strip it for you. Veneur will add it's own hostname as configured to metrics sent to Datadog.

## Expiration

Veneur expires all metrics on each flush. If a metric is no longer being sent (or is sent sparsely) Veneur will not send it as zeros! This was chosen because the combination of the approximation's features and the additional hysteresis imposed by *retaining* these approximations over time was deemed more complex than desirable.

{'fraction_non_alphanumeric': 0.06029882604055496, 'fraction_numerical': 0.004268943436499467, 'mean_word_length': 4.834993773349938, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2510681', 'n_tokens_mistral': 2643, 'n_tokens_neox': 2467, 'n_words': 1248}