<commit_before>It is recommended for all alerts to adhere to the follow guidelines:

1. [Keep conditions simple](#keep-it-simple).
2. [Alerts should be actionable](#actionable-alerts).
3. [Check for measured failure on critical paths](#check-for-measured-failure) rather than a lack of success.
4. [Alerts should not have special cases](#avoid-special-cases) for routine maintenance.
5. [Consider how the alert check can fail](#think-about-failure).
 
## Keep It Simple

When an alert triggers, it should be easy to understand why. Similarly, if an alert doesn't fire, then it should be easy to check and see what happened. The more complicated an alert condition becomes, the harder it is to understand and debug. 

It is recommended to keep alert rules as a simple expression with a threshold and number of occurrences. An example of this is the following rule:

```
CPU Usage > 80% for at least 5 minutes
``` 

Multiple signals should only be combined if it improves the effectiveness of the alert. For example, what is an appropriate threshold for the number of requests that have error responses? What happens to that threshold if your cluster auto-scales?  It is more effective to define the threshold as a percentage of total requests:

```
(Num Errors / Num Total) > 0.01 for at least 5 minutes
```

As a general rule, bias towards simplicity. If you are creating more complex expressions, then stop and think about why that complexity is needed. Are there other signals available that are easier to use? Can the application be changed so that it reports metrics which make it easier to diagnose? 

## Actionable Alerts

If an alert fires and sends a notification to users, someone should be motivated to investigate the problem. Alerts that are noisy or not actionable train people to ignore or filter out alert notifications. 

For cases where the response to an alert can be automated, such as terminating a bad instance, it shouldn't send out a notification unless there is a failure to perform the action. If you want a summary of cluster health, then use dashboards or reporting tools for this function; don't attempt to do this via alert notifications.

Alerts should check something important. To setup effective alerts, you need to understand the application and have ways to detect failures for critical functionality. Avoid general system-type alerts that won't be investigated. For example, should you alert on high CPU usage? If you have done squeeze testing and you have information to indicate how CPU usage impacts the application, then it can be useful and it will provide a way to know a problem is coming before it impacts clients of the service. If you do not have this knowledge, then your alert may be under-tuned, leading to noisy notifications that may be ignored.

## Check for Measured Failure

It is better to check for failures rather than trying to trigger based on an absence of information or a reduction in the amount of success.

### Absence of Information

A typical example of this is a process that runs over a longer time period. For example, suppose we have an application that updates a metadata cache once per day and it takes an hour to refresh. It is not recommended to send an event on refresh success and then configure alerts based on the absence of the success event. Design the signals so you have a clear way to understand what error conditions may be occurring on and then alert if there is a problem.

In this example, a better design would use a gauge that reports the loading time and a gauge that reports the age of the cache.  You can then add alerts when the gauges for these error conditions exceed unacceptable thresholds.

### Reduction in Success

Let's say we have a server that is taking traffic and we want to know if users are experiencing problems.  How should we go about this? It is often tempting to look for things like a drop in the number of successful requests, because this can be a generic catch-all for many types of problems.
<commit_msg>docs: discuss low volume conditions for alerting (#592)<commit_after>It is recommended for all alerts to adhere to the follow guidelines:

1. [Keep conditions simple](#keep-it-simple).
2. [Alerts should be actionable](#actionable-alerts).
3. [Check for measured failure on critical paths](#check-for-measured-failure) rather than a lack of success.
4. [Alerts should not have special cases](#avoid-special-cases) for routine maintenance.
5. [Consider how the alert check can fail](#think-about-failure).
 
## Keep It Simple

When an alert triggers, it should be easy to understand why. Similarly, if an alert doesn't fire, then it should be easy to check and see what happened. The more complicated an alert condition becomes, the harder it is to understand and debug. 

It is recommended to keep alert rules as a simple expression with a threshold and number of occurrences. An example of this is the following rule:

```
CPU Usage > 80% for at least 5 minutes
``` 

Multiple signals should only be combined if it improves the effectiveness of the alert. For example, what is an appropriate threshold for the number of requests that have error responses? What happens to that threshold if your cluster auto-scales?  It is more effective to define the threshold as a percentage of total requests:

```
(Num Errors / Num Total) > 0.01 for at least 5 minutes
```

In some cases, a low volume can make the percentages less meaningful and result in false positives. For example, if your daily traffic pattern follows a sine curve, then the troughs may not represent a meaningful error percentage. Another example might be during failover exercises, if traffic has been failed over to another cluster. One way to compensate for this is to check the failure rate and overall volume:

```
Percentage of Failures > X AND Volume > Y
```

As a general rule, bias towards simplicity. If you are creating more complex expressions, then stop and think about why that complexity is needed. Are there other signals available that are easier to use? Can the application be changed so that it reports metrics which make it easier to diagnose? 

## Actionable Alerts

If an alert fires and sends a notification to users, someone should be motivated to investigate the problem. Alerts that are noisy or not actionable train people to ignore or filter out alert notifications. 

For cases where the response to an alert can be automated, such as terminating a bad instance, it shouldn't send out a notification unless there is a failure to perform the action. If you want a summary of cluster health, then use dashboards or reporting tools for this function; don't attempt to do this via alert notifications.

Alerts should check something important. To setup effective alerts, you need to understand the application and have ways to detect failures for critical functionality. Avoid general system-type alerts that won't be investigated. For example, should you alert on high CPU usage? If you have done squeeze testing and you have information to indicate how CPU usage impacts the application, then it can be useful and it will provide a way to know a problem is coming before it impacts clients of the service. If you do not have this knowledge, then your alert may be under-tuned, leading to noisy notifications that may be ignored.

## Check for Measured Failure

It is better to check for failures rather than trying to trigger based on an absence of information or a reduction in the amount of success.

### Absence of Information

A typical example of this is a process that runs over a longer time period. For example, suppose we have an application that updates a metadata cache once per day and it takes an hour to refresh. It is not recommended to send an event on refresh success and then configure alerts based on the absence of the success event. Design the signals so you have a clear way to understand what error conditions may be occurring on and then alert if there is a problem.

In this example, a better design would use a gauge that reports the loading time and a gauge that reports the age of the cache.  You can then add alerts when the gauges for these error conditions exceed unacceptable thresholds.

### Reduction in Success

Let's say we have a server that is taking traffic and we want to know if users are experiencing problems.  How should we go about this? It is often tempting to look for things like a drop in the number of successful requests, because this can be a generic catch-all for many types of problems.

{'fraction_non_alphanumeric': 0.03555139749736873, 'fraction_numerical': 0.003157525435621565, 'mean_word_length': 4.368487131198996, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2490229', 'n_tokens_mistral': 1966, 'n_tokens_neox': 1835, 'n_words': 1403}