<commit_before>// Module included in the following assemblies:
//
// * logging/cluster-logging-deploy.adoc

[id="cluster-logging-deploy-storage-considerations_{context}"]
= Storage considerations for OpenShift Logging and {product-title}

////
An Elasticsearch index is a collection of primary shards and its corresponding replica
shards. This is how Elasticsearch implements high availability internally, so there
is little requirement to use hardware based mirroring RAID variants. RAID 0 can still
be used to increase overall disk performance.
////

A persistent volume is required for each Elasticsearch deployment configuration. On {product-title} this is achieved using
persistent volume claims.

The Elasticsearch Operator names the PVCs using the Elasticsearch resource name.

////
Below are capacity planning guidelines for {product-title} aggregate logging.

*Example scenario*

Assumptions:

. Which application: Apache
. Bytes per line: 256
. Lines per second load on application: 1
. Raw text data -> JSON

Baseline (256 characters per minute -> 15KB/min)

[cols="3,4",options="header"]
|===
|Logging pods
|Storage Throughput

|3 es
1 kibana
1 curator
1 fluentd
| 6 pods total: 90000 x 86400 = 7,7 GB/day

|3 es
1 kibana
1 curator
11 fluentd
| 16 pods total: 225000 x 86400 = 24,0 GB/day

|3 es
1 kibana
1 curator
20 fluentd
|25 pods total: 225000 x 86400 = 32,4 GB/day
|===


Calculating total logging throughput and disk space required for your {product-title} cluster requires knowledge of your applications. For example, if one of your
applications on average logs 10 lines-per-second, each 256 bytes-per-line,
calculate per-application throughput and disk space as follows:

----
 (bytes-per-line * (lines-per-second) = 2560 bytes per app per second
 (2560) * (number-of-pods-per-node,100) = 256,000 bytes per second per node
 256k * (number-of-nodes) = total logging throughput per cluster per second
----
////

Fluentd ships any logs from *systemd journal* and **/var/log/containers/*.log** to Elasticsearch.

Therefore, consider how much data you need in advance and that you are
aggregating application log data. Some Elasticsearch users have found that it
is necessary to keep absolute storage consumption around 50% and below 70% at all times. This
helps to avoid Elasticsearch becoming unresponsive during large merge
operations.

By default, at 85% Elasticsearch stops allocating new data to the node, at 90% Elasticsearch attempts to relocate
existing shards from that node to other nodes if possible. But if no nodes have free capacity below 85%, Elasticsearch effectively rejects creating new indices
and becomes RED.

[NOTE]
====
These low and high watermark values are Elasticsearch defaults in the current release. You can modify these values,
but you also must apply any modifications to the alerts also. The alerts are based
on these defaults.
====<commit_msg>RHDEVDOCS-2617 [DOCS] Alert manager modifications in cluster logging storage recommendations<commit_after>// Module included in the following assemblies:
//
// * logging/cluster-logging-deploy.adoc

[id="cluster-logging-deploy-storage-considerations_{context}"]
= Storage considerations for OpenShift Logging and {product-title}

////
An Elasticsearch index is a collection of primary shards and their corresponding replica shards. This is how Elasticsearch implements high availability internally, so there is little requirement to use hardware based mirroring RAID variants. RAID 0 can still be used to increase overall disk performance.
////

A persistent volume is required for each Elasticsearch deployment configuration. On {product-title} this is achieved using persistent volume claims.

The Elasticsearch Operator names the PVCs using the Elasticsearch resource name.

////
Below are capacity planning guidelines for {product-title} aggregate logging.

*Example scenario*

Assumptions:

. Which application: Apache
. Bytes per line: 256
. Lines per second load on application: 1
. Raw text data -> JSON

Baseline (256 characters per minute -> 15KB/min)

[cols="3,4",options="header"]
|===
|Logging pods
|Storage Throughput

|3 es
1 kibana
1 curator
1 fluentd
| 6 pods total: 90000 x 86400 = 7,7 GB/day

|3 es
1 kibana
1 curator
11 fluentd
| 16 pods total: 225000 x 86400 = 24,0 GB/day

|3 es
1 kibana
1 curator
20 fluentd
|25 pods total: 225000 x 86400 = 32,4 GB/day
|===


Calculating the total logging throughput and disk space required for your {product-title} cluster requires knowledge of your applications. For example, if one of your applications on average logs 10 lines-per-second, each 256 bytes-per-line, calculate per-application throughput and disk space as follows:

----
 (bytes-per-line * (lines-per-second) = 2560 bytes per app per second
 (2560) * (number-of-pods-per-node,100) = 256,000 bytes per second per node
 256k * (number-of-nodes) = total logging throughput per cluster per second
----
////

Fluentd ships any logs from *systemd journal* and **/var/log/containers/*.log** to Elasticsearch.

Elasticsearch requires sufficient memory to perform large merge operations. If it does not have enough memory, it becomes unresponsive. To avoid this problem, evaluate how much application log data you need, and allocate approximately double that amount of free storage capacity.

By default, when storage capacity is 85% full, Elasticsearch stops allocating new data to the node. At 90%, Elasticsearch attempts to relocate existing shards from that node to other nodes if possible. But if no nodes have a free capacity below 85%, Elasticsearch effectively rejects creating new indices and becomes RED.

[NOTE]
====
These low and high watermark values are Elasticsearch defaults in the current release. You can modify these default values. Although the alerts use the same default values, you cannot change these values in the alerts.
====
{'fraction_non_alphanumeric': 0.0704778156996587, 'fraction_numerical': 0.03617747440273038, 'mean_word_length': 4.477570093457944, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4438972', 'n_tokens_mistral': 1720, 'n_tokens_neox': 1529, 'n_words': 784}