<commit_before>---
title: Outlier Detection
kind: documentation
sidebar:
  nav:
    - header: Outlier Detection
    - text: Show Outliers in Dashboards
      href: "#dashboards"
    - text: Alert on Outliers
      href: "#alerts"
    - text: Algorithms & Parameters
      href: "#algorithms"
---

Outlier Detection is an algorithmic feature that allows you to detect when some members of a group are behaving strangely compared to the others. For example, you could detect that one web server in a pool is processing an unusual number of requests, and hence should be a target for replacement. Or, you could get an early warning that significantly more 500s are happening in one AWS Availability Zone (AZ) than the others, which might indicate an issue brewing in that AZ.

<img src="/static/images/outliers/outliers-metric-alert.png" style="width:100%; border:1px solid #777777"/>

## How to Use Outlier Detection on Your Data

We’ve added a new query function called `outliers` to our query language. This function will return the usual results but outlier series will be marked.

You can use this function to display and alert on outliers in your data. To try it out, you’ll first need a metric for which a group of hosts (or availability zones, partitions, etc) should exhibit uniform behavior. For the function to work, be sure that there are at least 3 or more members in the group. Given that, here are two ways to use outlier detection on that group.


### 1. Show Outliers in Dashboards or Screenboards
{: #dashboards}

For example, here is a graph of gunicorn requests by host with outlier detection enabled:

<img src="/static/images/outliers/outliers-graph-dbscan-gunicorn.png" style="width:100%; border:1px solid #777777"/>

You can see that one of the series is an outlier: it is handling significantly lower traffic than the others for the time window in question.

To set up an outlier detection graph for your data you add a metric to the graph showing all series in the groups. You apply the outlier detection algorithm by adding `outliers` function on your data. After applying the function, outlier series will be colored with a bold, warm palette, while all other series will be colored with a lightweight, greyscale color palette.

To do so, create a new timeseries graph on your dashboard with your chosen metric. Your screen should look like:

<img src="/static/images/outliers/outliers-dash-choose-metrics-newer.png" style="width:100%; border:1px solid #777777"/>

Now, click on the + icon (Add functions and modifiers) on the right side of the second metrics line. In the "Modify your query" box, choose the "outliers" function:

<img src="/static/images/outliers/outliers-function-selector-newer.png" style="width:25%; border:1px solid #777777"/>

This will add the outliers function to your graph, and you’ll see any outliers in the group highlighted in bold, warm colors.

<img src="/static/images/outliers/outliers-algorithm-annotated-newer.png" style="width:100%; border:1px solid #777777"/>

There are several outlier detection algorithms you can choose. The default algorithm (DBSCAN) and parameter values should work for most scenarios. However, if you see too many or too few outliers identified, you can tune the algorithm or try an alternate algorithm. To learn more, see the "Outlier Algorithms and Parameters" section below.


### 2. Alert on Outliers
{: #alerts}

You can also define a monitor to alert when an outlier is detected in an important group.

<img src="/static/images/outliers/outliers-alert-snapshot.png" style="width:100%; border:1px solid #777777"/>

For example, to alert when a Cassandra host is abnormally loaded compared to the rest of the group, we’d [add a new outlier monitor](https://app.datadoghq.com/monitors#create/algorithm) for our metric:

<img src="/static/images/outliers/outliers-new-monitor-define-metric.png" style="width:100%; border:1px solid #777777"/>

You will select the metric and scope as with other metric-based monitors.

In the alert conditions you will select the grouping and timeframe.

You can also optionally select an algorithm to use for outlier detection. By default we have chosen DBSCAN with an alpha value of 3 because this works for many cases. More information about the outlier functions and their parameters is available below.
<commit_msg>[outliers] changing lingering alpha to tolerance<commit_after>---
title: Outlier Detection
kind: documentation
sidebar:
  nav:
    - header: Outlier Detection
    - text: Show Outliers in Dashboards
      href: "#dashboards"
    - text: Alert on Outliers
      href: "#alerts"
    - text: Algorithms & Parameters
      href: "#algorithms"
---

Outlier Detection is an algorithmic feature that allows you to detect when some members of a group are behaving strangely compared to the others. For example, you could detect that one web server in a pool is processing an unusual number of requests, and hence should be a target for replacement. Or, you could get an early warning that significantly more 500s are happening in one AWS Availability Zone (AZ) than the others, which might indicate an issue brewing in that AZ.

<img src="/static/images/outliers/outliers-metric-alert.png" style="width:100%; border:1px solid #777777"/>

## How to Use Outlier Detection on Your Data

We’ve added a new query function called `outliers` to our query language. This function will return the usual results but outlier series will be marked.

You can use this function to display and alert on outliers in your data. To try it out, you’ll first need a metric for which a group of hosts (or availability zones, partitions, etc) should exhibit uniform behavior. For the function to work, be sure that there are at least 3 or more members in the group. Given that, here are two ways to use outlier detection on that group.


### 1. Show Outliers in Dashboards or Screenboards
{: #dashboards}

For example, here is a graph of gunicorn requests by host with outlier detection enabled:

<img src="/static/images/outliers/outliers-graph-dbscan-gunicorn.png" style="width:100%; border:1px solid #777777"/>

You can see that one of the series is an outlier: it is handling significantly lower traffic than the others for the time window in question.

To set up an outlier detection graph for your data you add a metric to the graph showing all series in the groups. You apply the outlier detection algorithm by adding `outliers` function on your data. After applying the function, outlier series will be colored with a bold, warm palette, while all other series will be colored with a lightweight, greyscale color palette.

To do so, create a new timeseries graph on your dashboard with your chosen metric. Your screen should look like:

<img src="/static/images/outliers/outliers-dash-choose-metrics-newer.png" style="width:100%; border:1px solid #777777"/>

Now, click on the + icon (Add functions and modifiers) on the right side of the second metrics line. In the "Modify your query" box, choose the "outliers" function:

<img src="/static/images/outliers/outliers-function-selector-newer.png" style="width:25%; border:1px solid #777777"/>

This will add the outliers function to your graph, and you’ll see any outliers in the group highlighted in bold, warm colors.

<img src="/static/images/outliers/outliers-algorithm-annotated-newer.png" style="width:100%; border:1px solid #777777"/>

There are several outlier detection algorithms you can choose. The default algorithm (DBSCAN) and parameter values should work for most scenarios. However, if you see too many or too few outliers identified, you can tune the algorithm or try an alternate algorithm. To learn more, see the "Outlier Algorithms and Parameters" section below.


### 2. Alert on Outliers
{: #alerts}

You can also define a monitor to alert when an outlier is detected in an important group.

<img src="/static/images/outliers/outliers-alert-snapshot.png" style="width:100%; border:1px solid #777777"/>

For example, to alert when a Cassandra host is abnormally loaded compared to the rest of the group, we’d [add a new outlier monitor](https://app.datadoghq.com/monitors#create/algorithm) for our metric:

<img src="/static/images/outliers/outliers-new-monitor-define-metric.png" style="width:100%; border:1px solid #777777"/>

You will select the metric and scope as with other metric-based monitors.

In the alert conditions you will select the grouping and timeframe.

You can also optionally select an algorithm to use for outlier detection. By default we have chosen DBSCAN with a tolerance value of 3 because this works for many cases. More information about the outlier functions and their parameters is available below.

{'fraction_non_alphanumeric': 0.06836430373938977, 'fraction_numerical': 0.017435191557696718, 'mean_word_length': 4.511378002528445, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1278366', 'n_tokens_mistral': 2447, 'n_tokens_neox': 2221, 'n_words': 1215}