<commit_before># Data knobs

See [the example config](https://github.com/raintank/metrictank/blob/master/metrictank-sample.ini) for an overview and basic explanation of what the config values are.
Some of the values related to chunking and compression are a bit harder to tune, so this article will explain in more detail.

## Compression tips

* values that never - or infrequently - change compress extremely well, so are very cheap to track and store.
* pay attention to your timestamps, make sure they are evenly spaced. That compresses better.
* storing values as integers (or more precisely: floats without decimal numbers) compresses very well.
  So you best store the numbers with the same unit as what your precision is.
  E.g. let's say you measure latencies such as 0.035 seconds (3 decimals precision, e.g. ms precision), it's better to
  track that as the number 35 (milliseconds) instead of 0.035 (seconds).

For more details, see the [go-tsz eval program](https://github.com/dgryski/go-tsz/tree/master/eval) or the 
[results table](https://raw.githubusercontent.com/dgryski/go-tsz/master/eval/eval-results.png)

## Chunk sizing and num chunks to keep in memory

### Basic guideline

`chunkspan` is how long of a timeframe should be covered by your chunks. E.g. you could store anywhere between 1 second to 24 hours worth of data in a chunk (chunks for each raw metric).
`numchunks` is simply how many chunks should be retained in RAM. (for each raw metric)

figuring out optimal configuration for the `chunkspan` and `numchunks` is not trivial.
The standard recommendation is 120 points per chunk and keep at least as much in RAM as what your commonly query for (+1 extra chunk, see below)

E.g. if your most common interval is 10s and most of your dashboards query for 2h worth of data, then the recommendation is:
```
chunkspan = 20min
numchunks = 1
```

20min because 120 points per chunk every 10 seconds is 1200 seconds or 20 minutes.
If you expected 6 chunks (20min * 6 = 2h), the answer is that you always need 1 extra chunk,
because the current chunk is typically incomplete and only covers a fraction of the ongoing 20min timeslot,
so you should always make sure to cover the requirements of one extra chunkspan.

Note:
* `chunkspan` and `numchunks` are currently global variables, which can't be finetuned on a per-metric or per-category level.
* when defining consolidation (rollups), you can specify custom chunkspans and numchunks for each rollup setting.  As rollups will have more time between points, it makes sense to choose longer chunkspans for rollups.

### Additional factors

Several factors come into play that may affect the above recommendation:

#### Valid chunk spans

Chunkspans can be set to one of the following:
```
1sec, 5sec, 10sec, 15sec, 20sec, 30sec, 60sec, 90sec, 2min, 3min, 5min, 10min, 15min, 20min,
30min, 45min, 1h, 90min, 2h, 150min, 3h, 4h, 5h, 6h, 7h, 8h, 9h, 10h, 12h, 15h, 18h, 24h
```
This list can be extended in the future.

#### Rollups remove the need to keep large number of higher resolution chunks

If you roll-up data for archival storage, those chunks will also be in memory as per your configuration.
Querying for large timeframes may use the consolidated chunks in RAM, and keeping
extra raw (or higher-resolution) data in RAM becomes pointless, putting an upper bound on how many chunks to keep.  
See [Consolidation](https://github.com/raintank/metrictank/blob/master/docs/consolidation.md)


#### Compression efficiency

The more points are contained within a chunk, the more efficiently the compression can work. This is very noticeable
until about 120 points per chunk, at which point the improvement becomes less relevant.
For more details, see the [go-tsz eval program](https://github.com/dgryski/go-tsz/tree/master/eval) or the 
[results table](https://raw.githubusercontent.com/dgryski/go-tsz/master/eval/eval-results.png)


#### Garbage collection pressure

If garbage collection is a problem (which can be analyzed with the metrictank grafana dashboard), then this can be reduced by picking longer chunks and keeping fewer of them.
The more chunks you have, the more they need to be scanned by the Go garbage collector.<commit_msg>some docs updates<commit_after># Data knobs

See [the example config](https://github.com/raintank/metrictank/blob/master/metrictank-sample.ini) for an overview and basic explanation of what the config values are.
Some of the values related to chunking and compression are a bit harder to tune, so this article will explain in more detail.

## Compression tips

* values that never - or infrequently - change compress extremely well, so are very cheap to track and store.
* pay attention to your timestamps, make sure they are evenly spaced. That compresses better.
* storing values as integers (or more precisely: floats without decimal numbers) compresses very well.
  So you best store the numbers with the same unit as what your precision is.
  E.g. let's say you measure latencies such as 0.035 seconds (3 decimals precision, e.g. ms precision), it's better to
  track that as the number 35 (milliseconds) instead of 0.035 (seconds).

For more details, see the [go-tsz eval program](https://github.com/dgryski/go-tsz/tree/master/eval) or the 
[results table](https://raw.githubusercontent.com/dgryski/go-tsz/master/eval/eval-results.png)

## Chunk sizing and num chunks to keep in memory

### Basic guideline

`chunkspan` is how long of a timeframe should be covered by your chunks. E.g. you could store anywhere between 1 second to 24 hours worth of data in a chunk (chunks for each raw metric).
`numchunks` is simply how many chunks should be retained in RAM (for each raw metric).

Figuring out the optimal configuration for the `chunkspan` and `numchunks` is not trivial.
The standard recommendation is at least 120 points per chunk and at least 1 chunk. Once data is older than `chunkspan * numchunks` it will be retained in memory by the
[chunk-cache](#chunk-cache) if it is queried frequently.

E.g. if your most common interval is 10s and your dashboards repeatedly query the same metrics, then the recommendation is:
```
chunkspan = 20min
numchunks = 1
```

20min because 120 points per chunk every 10 seconds is 1200 seconds or 20 minutes.
If you your dashboards repeatedly query the same metrics then it is safe to assume that the hot data which is older than `20min` will be served by the [chunk-cache](#chunk-cache) most of the time.

Note:
* `chunkspan` and `numchunks` are currently global variables, which can't be finetuned on a per-metric or per-category level.
* when defining consolidation (rollups), you can specify custom chunkspans and numchunks for each rollup setting.  As rollups will have more time between points, it makes sense to choose longer chunkspans for rollups.

### Additional factors

Several factors come into play that may affect the above recommendation:

#### Valid chunk spans

Chunkspans can be set to one of the following:
```
1sec, 5sec, 10sec, 15sec, 20sec, 30sec, 60sec, 90sec, 2min, 3min, 5min, 10min, 15min, 20min,
30min, 45min, 1h, 90min, 2h, 150min, 3h, 4h, 5h, 6h, 7h, 8h, 9h, 10h, 12h, 15h, 18h, 24h
```
This list can be extended in the future.

#### Chunk Cache

If data is fetched from Cassandra it gets added into an internal caching layer that we call the `chunk-cache`, as the name implies this caches chunk by chunk.
The chunk cache has a configurable [maximum size](https://github.com/raintank/metrictank/blob/master/docs/config.md#chunk-cache),
within that size it tries to always keep the most often queried data by using an LRU mechanism that evicts the Least Recently Used chunks once the
total size of the chunk cache reaches the defined maximum.
The efficiency of the chunk cache largely depends on the common query patterns and the configured `max-size` value.
If a small number of metrics gets queried often, then the efficiency of the chunk cache will be rather high because it can serve most requests out of it's memory.
On the other hand, if most queries involve metrics that have not been queried for a long time and if they are only queried a small number of times,
then Metrictank will need to fallback to Cassandra often and the chunk cache efficiency will decrease.

#### Rollups remove the need to keep large number of higher resolution chunks

If you roll-up data for archival storage, those chunks will also be in memory as per your configuration.
Querying for large timeframes may use the consolidated chunks in RAM, and keeping
extra raw (or higher-resolution) data in RAM becomes pointless, putting an upper bound on how many chunks to keep.  
See [Consolidation](https://github.com/raintank/metrictank/blob/master/docs/consolidation.md)


#### Compression efficiency

The more points are contained within a chunk, the more efficiently the compression can work. This is very noticeable
until about 120 points per chunk, at which point the improvement becomes less relevant.
For more details, see the [go-tsz eval program](https://github.com/dgryski/go-tsz/tree/master/eval) or the 
[results table](https://raw.githubusercontent.com/dgryski/go-tsz/master/eval/eval-results.png)


#### Garbage collection pressure

If garbage collection is a problem (which can be analyzed with the metrictank grafana dashboard), then this can be reduced by picking longer chunks and keeping fewer of them.
The more chunks you have, the more they need to be scanned by the Go garbage collector.
{'fraction_non_alphanumeric': 0.06435956952943658, 'fraction_numerical': 0.020151930787085885, 'mean_word_length': 4.469705712637046, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 13, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1156471', 'n_tokens_mistral': 2750, 'n_tokens_neox': 2505, 'n_words': 1385}