<commit_before>[![deb](https://img.shields.io/badge/deb-packagecloud.io-844fec.svg)](https://packagecloud.io/go-graphite/stable)
[![rpm](https://img.shields.io/badge/rpm-packagecloud.io-844fec.svg)](https://packagecloud.io/go-graphite/stable)

Table of Contents
=================

* [Features](#features)
* [Performance](#performance)
* [Installation](#installation)
* [Configuration](#configuration)
  * [OS tuning](#os-tuning)
* [Reported stats](#reported-stats)
* [Changelog](#changelog)
  * [master](#master)


go-carbon [![Build Status](https://travis-ci.org/go-graphite/go-carbon.svg?branch=master)](https://travis-ci.org/go-graphite/go-carbon)
============

Golang implementation of Graphite/Carbon server with classic architecture: Agent -> Cache -> Persister

![Architecture](doc/design.png)

## Features
- Receive metrics from TCP and UDP ([plaintext protocol](http://graphite.readthedocs.org/en/latest/feeding-carbon.html#the-plaintext-protocol))
- Receive metrics with [Pickle protocol](http://graphite.readthedocs.org/en/latest/feeding-carbon.html#the-pickle-protocol) (TCP only)
- Receive metrics from HTTP
- Receive metrics from Apache Kafka
- [storage-schemas.conf](http://graphite.readthedocs.org/en/latest/config-carbon.html#storage-schemas-conf)
- [storage-aggregation.conf](http://graphite.readthedocs.org/en/latest/config-carbon.html#storage-aggregation-conf)
- Carbonlink (requests to cache from graphite-web)
- Carbonlink-like GRPC api
- Logging with rotation support (reopen log if it moves)
- Many persister workers (using many cpu cores)
- Run as daemon
- Optional dump/restore restart on `USR2` signal (config `dump` section): stop persister, start write new data to file, dump cache to file, stop all (and restore from files after next start)
- Reload some config options without restart (HUP signal):
  - `whisper` section of main config, `storage-schemas.conf` and `storage-aggregation.conf`
  - `graph-prefix`, `metric-interval`, `metric-endpoint`, `max-cpu` from `common` section
  - `dump` section

## Performance

Faster than default carbon. In all conditions :) How much faster depends on server hardware, storage-schemas, etc.

The result of replacing "carbon" to "go-carbon" on a server with a load up to 900 thousand metric per minute:

![Success story](doc/success1.png)

There were some efforts to find out maximum possible performance of go-carbon on a hardware (2xE5-2620v3, 128GB RAM, local SSDs).

The result of that effort (in points per second):

![Performance](doc/performance.png)

Stable performance was around 950k points per second with short-term peak performance of 1.2M points/sec.

### Efficient metric namespace patterns for trie index

Putting the most common namespaces at the beginning of the metric name could be beneficial for scaling. This doesn't affect the performance of trigram index. But when you decide to switch to trie index for serving higher number metrics in a single go-carbon instance, your query would be more efficient. At the same time, this naming pattern could lead to better metric namespace hierarchy.

For example: querying `sys.cpu.loadavg.app.host-0001` would be faster than querying `sys.app.host-0001.cpu.loadavg` using trie index. Especially when you have tens of thousands hosts (host-0001, ..., host-9999), they all share the same prefix of `sys.cpu.loadavg.app.host-` in trie index, and they are compared only once during query. So this patterns leads to better memory usage and query performance when using trie+nfa/dfa index.

More details could be found in this blog: [To glob 10M metrics: Trie * DFA = Tree² for Go-Carbon](https://www.xhu.buzz/to-glob-10m-metrics-using-trie-and-dfa/).

## Installation
Use binary packages from [releases page](https://github.com/lomik/go-carbon/releases) or build manually (requires golang 1.8+):
```
# build binary
git clone https://github.com/go-graphite/go-carbon.git
cd go-carbon
make
```

We are using <a href="https://packagecloud.io/"><img alt="Private Maven, RPM, DEB, PyPi and RubyGem Repository | packagecloud" height="46" src="https://packagecloud.io/images/packagecloud-badge.png" width="158" /></a> to host our packages!

At this moment we are building deb and rpm packages for i386, amd64 and arm64 architectures. Installation guides are available on packagecloud (see the links below).

Stable versions: [Stable repo](https://packagecloud.io/go-graphite/stable/install)

Autobuilds (master, might be unstable): [Autobuild repo](https://packagecloud.io/go-graphite/autobuilds/install)

We're uploading Docker images to [Docker Hub](https://hub.docker.com/repository/docker/gographite/go-carbon/general) and [ghcr.io](https://github.com/go-graphite/go-carbon/pkgs/container/go-carbon)

Also, you can download test packages from build artifacts: 
<commit_msg>readme about artifact download<commit_after>[![deb](https://img.shields.io/badge/deb-packagecloud.io-844fec.svg)](https://packagecloud.io/go-graphite/stable)
[![rpm](https://img.shields.io/badge/rpm-packagecloud.io-844fec.svg)](https://packagecloud.io/go-graphite/stable)

Table of Contents
=================

* [Features](#features)
* [Performance](#performance)
* [Installation](#installation)
* [Configuration](#configuration)
  * [OS tuning](#os-tuning)
* [Reported stats](#reported-stats)
* [Changelog](#changelog)
  * [master](#master)


go-carbon [![Build Status](https://travis-ci.org/go-graphite/go-carbon.svg?branch=master)](https://travis-ci.org/go-graphite/go-carbon)
============

Golang implementation of Graphite/Carbon server with classic architecture: Agent -> Cache -> Persister

![Architecture](doc/design.png)

## Features
- Receive metrics from TCP and UDP ([plaintext protocol](http://graphite.readthedocs.org/en/latest/feeding-carbon.html#the-plaintext-protocol))
- Receive metrics with [Pickle protocol](http://graphite.readthedocs.org/en/latest/feeding-carbon.html#the-pickle-protocol) (TCP only)
- Receive metrics from HTTP
- Receive metrics from Apache Kafka
- [storage-schemas.conf](http://graphite.readthedocs.org/en/latest/config-carbon.html#storage-schemas-conf)
- [storage-aggregation.conf](http://graphite.readthedocs.org/en/latest/config-carbon.html#storage-aggregation-conf)
- Carbonlink (requests to cache from graphite-web)
- Carbonlink-like GRPC api
- Logging with rotation support (reopen log if it moves)
- Many persister workers (using many cpu cores)
- Run as daemon
- Optional dump/restore restart on `USR2` signal (config `dump` section): stop persister, start write new data to file, dump cache to file, stop all (and restore from files after next start)
- Reload some config options without restart (HUP signal):
  - `whisper` section of main config, `storage-schemas.conf` and `storage-aggregation.conf`
  - `graph-prefix`, `metric-interval`, `metric-endpoint`, `max-cpu` from `common` section
  - `dump` section

## Performance

Faster than default carbon. In all conditions :) How much faster depends on server hardware, storage-schemas, etc.

The result of replacing "carbon" to "go-carbon" on a server with a load up to 900 thousand metric per minute:

![Success story](doc/success1.png)

There were some efforts to find out maximum possible performance of go-carbon on a hardware (2xE5-2620v3, 128GB RAM, local SSDs).

The result of that effort (in points per second):

![Performance](doc/performance.png)

Stable performance was around 950k points per second with short-term peak performance of 1.2M points/sec.

### Efficient metric namespace patterns for trie index

Putting the most common namespaces at the beginning of the metric name could be beneficial for scaling. This doesn't affect the performance of trigram index. But when you decide to switch to trie index for serving higher number metrics in a single go-carbon instance, your query would be more efficient. At the same time, this naming pattern could lead to better metric namespace hierarchy.

For example: querying `sys.cpu.loadavg.app.host-0001` would be faster than querying `sys.app.host-0001.cpu.loadavg` using trie index. Especially when you have tens of thousands hosts (host-0001, ..., host-9999), they all share the same prefix of `sys.cpu.loadavg.app.host-` in trie index, and they are compared only once during query. So this patterns leads to better memory usage and query performance when using trie+nfa/dfa index.

More details could be found in this blog: [To glob 10M metrics: Trie * DFA = Tree² for Go-Carbon](https://www.xhu.buzz/to-glob-10m-metrics-using-trie-and-dfa/).

## Installation
Use binary packages from [releases page](https://github.com/lomik/go-carbon/releases) or build manually (requires golang 1.8+):
```
# build binary
git clone https://github.com/go-graphite/go-carbon.git
cd go-carbon
make
```

We are using <a href="https://packagecloud.io/"><img alt="Private Maven, RPM, DEB, PyPi and RubyGem Repository | packagecloud" height="46" src="https://packagecloud.io/images/packagecloud-badge.png" width="158" /></a> to host our packages!

At this moment we are building deb and rpm packages for i386, amd64 and arm64 architectures. Installation guides are available on packagecloud (see the links below).

Stable versions: [Stable repo](https://packagecloud.io/go-graphite/stable/install)

Autobuilds (master, might be unstable): [Autobuild repo](https://packagecloud.io/go-graphite/autobuilds/install)

We're uploading Docker images to [Docker Hub](https://hub.docker.com/repository/docker/gographite/go-carbon/general) and [ghcr.io](https://github.com/go-graphite/go-carbon/pkgs/container/go-carbon)

Also, you can download test packages from build artifacts: Go to list of [test runs](https://github.com/go-graphite/go-carbon/actions/workflows/tests.yml), click on PR name, and click on *"packages-^1"* under *"Artifact"* section.

{'fraction_non_alphanumeric': 0.12351859419697589, 'fraction_numerical': 0.012362076011442583, 'mean_word_length': 4.972544234289201, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 13, 'https://': 31, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6139825', 'n_tokens_mistral': 3107, 'n_tokens_neox': 2863, 'n_words': 954}