<commit_before>[role="xpack"]
[[ml-configuring-aggregation]]
=== Aggregating data for faster performance

By default, {dfeeds} fetch data from {es} using search and scroll requests.
It can be significantly more efficient, however, to aggregate data in {es}
and to configure your {anomaly-jobs} to analyze aggregated data.

One of the benefits of aggregating data this way is that {es} automatically
distributes these calculations across your cluster. You can then feed this
aggregated data into the {ml-features} instead of raw results, which
reduces the volume of data that must be considered while detecting anomalies.

TIP: If you use a terms aggregation and the cardinality of a term is high, the
aggregation might not be effective and you might want to just use the default
search and scroll behavior.

[discrete]
[[aggs-limits-dfeeds]]
==== Requirements and limitations

There are some limitations to using aggregations in {dfeeds}. Your aggregation
must include a `date_histogram` aggregation, which in turn must contain a `max`
aggregation on the time field. This requirement ensures that the aggregated data
is a time series and the timestamp of each bucket is the time of the last record
in the bucket.

IMPORTANT: The name of the aggregation and the name of the field that the agg 
operates on need to match, otherwise the aggregation doesn't work. For example, 
if you use a `max` aggregation on a time field called `responsetime`, the name 
of the aggregation must be also `responsetime`.

You must also consider the interval of the date histogram aggregation carefully.
The bucket span of your {anomaly-job} must be divisible by the value of the
`calendar_interval` or `fixed_interval` in your aggregation (with no remainder).
If you specify a `frequency` for your {dfeed}, it must also be divisible by this
interval. {anomaly-jobs-cap} cannot use date histograms with an interval 
measured in months because the length of the month is not fixed. {dfeeds-cap} 
tolerate weeks or smaller units.

TIP: As a rule of thumb, if your detectors use <<ml-metric-functions,metric>> or
<<ml-sum-functions,sum>> analytical functions, set the date histogram
aggregation interval to a tenth of the bucket span. This suggestion creates
finer, more granular time buckets, which are ideal for this type of analysis. If
your detectors use <<ml-count-functions,count>> or <<ml-rare-functions,rare>>
functions, set the interval to the same value as the bucket span.


[discrete]
[[aggs-include-jobs]]
==== Including aggregations in {anomaly-jobs}

When you create or update an {anomaly-job}, you can include the names of
aggregations, for example:

[source,console]
----------------------------------
PUT _ml/anomaly_detectors/farequote
{
  "analysis_config": {
    "bucket_span": "60m",
    "detectors": [{
      "function": "mean",
      "field_name": "responsetime",  <1>
      "by_field_name": "airline"  <1>
    }],
    "summary_count_field_name": "doc_count"
  },
  "data_description": {
    "time_field":"time"  <1>
  }
}
----------------------------------
// TEST[skip:setup:farequote_data]

<1> In this example, the `airline`, `responsetime`, and `time` fields are
aggregations. Only the aggregated fields defined in the `analysis_config` object 
are analyzed by the {anomaly-job}.

NOTE: When the `summary_count_field_name` property is set to a non-null value,
the job expects to receive aggregated input. The property must be set to the
name of the field that contains the count of raw data points that have been
aggregated. It applies to all detectors in the job.

The aggregations are defined in the {dfeed} as follows:

[source,console]
----------------------------------
PUT _ml/datafeeds/datafeed-farequote
{
  "job_id":"farequote",
  "indices": ["farequote"],
  "aggregations": {
    "buckets": {
      "date_histogram": {
        "field": "time",
        "fixed_interval": "360s",
        "time_zone": "UTC"
      },
      "aggregations": {
        "time": {  <1>
          "max": {"field": "time"}
        },
        "airline": {  <2>
          "terms": {
            "field": "airline",
            "size": 100
          },
          "aggregations": {
            "responsetime": {  <3>
              "avg": {
                "field": "responsetime"
              }
            }
          }
        }
      }
    }
  }
}
----------------------------------
// TEST[skip:setup:farequote_job]

<1> In this example, the aggregations have names that match the fields that they
operate on. That is to say, the `max` aggregation is named `time` and its
field also needs to be `time`.
<2> Likewise, the `term` aggregation is named `airline` and its field is also 
named `airline`.
<3> Likewise, the `avg` aggregation is named `responsetime` and its field is 
also named `responsetime`.

Your {dfeed} can contain multiple aggregations, but only the ones with names 
that match values in the job configuration are fed to the job.


[discrete]
[[aggs-dfeeds]]
==== Nested aggregations in {dfeeds}

{dfeeds-cap} support complex nested aggregations. This example uses the 
`derivative` pipeline aggregation to find the first order derivative of the 
counter `system.network.out.bytes` for each value of the field `beat.name`.

[source,js]
----------------------------------
"aggregations": {
  "beat.name": {
    "terms": {
      "field": "beat.name"
    },
    "aggregations": {
      "buckets": {
        "date_histogram": {
          "field": "@timestamp",
          "fixed_interval": "5m"
        },
        "aggregations": {
          "@timestamp": {
            "max": {
              "field": "@timestamp"
            }<commit_msg>[DOCS] Makes the footnotes less verbose in configuring aggs page. (#55857)<commit_after>[role="xpack"]
[[ml-configuring-aggregation]]
=== Aggregating data for faster performance

By default, {dfeeds} fetch data from {es} using search and scroll requests.
It can be significantly more efficient, however, to aggregate data in {es}
and to configure your {anomaly-jobs} to analyze aggregated data.

One of the benefits of aggregating data this way is that {es} automatically
distributes these calculations across your cluster. You can then feed this
aggregated data into the {ml-features} instead of raw results, which
reduces the volume of data that must be considered while detecting anomalies.

TIP: If you use a terms aggregation and the cardinality of a term is high, the
aggregation might not be effective and you might want to just use the default
search and scroll behavior.

[discrete]
[[aggs-limits-dfeeds]]
==== Requirements and limitations

There are some limitations to using aggregations in {dfeeds}. Your aggregation
must include a `date_histogram` aggregation, which in turn must contain a `max`
aggregation on the time field. This requirement ensures that the aggregated data
is a time series and the timestamp of each bucket is the time of the last record
in the bucket.

IMPORTANT: The name of the aggregation and the name of the field that the agg 
operates on need to match, otherwise the aggregation doesn't work. For example, 
if you use a `max` aggregation on a time field called `responsetime`, the name 
of the aggregation must be also `responsetime`.

You must also consider the interval of the date histogram aggregation carefully.
The bucket span of your {anomaly-job} must be divisible by the value of the
`calendar_interval` or `fixed_interval` in your aggregation (with no remainder).
If you specify a `frequency` for your {dfeed}, it must also be divisible by this
interval. {anomaly-jobs-cap} cannot use date histograms with an interval 
measured in months because the length of the month is not fixed. {dfeeds-cap} 
tolerate weeks or smaller units.

TIP: As a rule of thumb, if your detectors use <<ml-metric-functions,metric>> or
<<ml-sum-functions,sum>> analytical functions, set the date histogram
aggregation interval to a tenth of the bucket span. This suggestion creates
finer, more granular time buckets, which are ideal for this type of analysis. If
your detectors use <<ml-count-functions,count>> or <<ml-rare-functions,rare>>
functions, set the interval to the same value as the bucket span.


[discrete]
[[aggs-include-jobs]]
==== Including aggregations in {anomaly-jobs}

When you create or update an {anomaly-job}, you can include the names of
aggregations, for example:

[source,console]
----------------------------------
PUT _ml/anomaly_detectors/farequote
{
  "analysis_config": {
    "bucket_span": "60m",
    "detectors": [{
      "function": "mean",
      "field_name": "responsetime",  <1>
      "by_field_name": "airline"  <1>
    }],
    "summary_count_field_name": "doc_count"
  },
  "data_description": {
    "time_field":"time"  <1>
  }
}
----------------------------------
// TEST[skip:setup:farequote_data]

<1> The `airline`, `responsetime`, and `time` fields are aggregations. Only the 
aggregated fields defined in the `analysis_config` object are analyzed by the 
{anomaly-job}.

NOTE: When the `summary_count_field_name` property is set to a non-null value,
the job expects to receive aggregated input. The property must be set to the
name of the field that contains the count of raw data points that have been
aggregated. It applies to all detectors in the job.

The aggregations are defined in the {dfeed} as follows:

[source,console]
----------------------------------
PUT _ml/datafeeds/datafeed-farequote
{
  "job_id":"farequote",
  "indices": ["farequote"],
  "aggregations": {
    "buckets": {
      "date_histogram": {
        "field": "time",
        "fixed_interval": "360s",
        "time_zone": "UTC"
      },
      "aggregations": {
        "time": {  <1>
          "max": {"field": "time"}
        },
        "airline": {  <2>
          "terms": {
            "field": "airline",
            "size": 100
          },
          "aggregations": {
            "responsetime": {  <3>
              "avg": {
                "field": "responsetime"
              }
            }
          }
        }
      }
    }
  }
}
----------------------------------
// TEST[skip:setup:farequote_job]

<1> The aggregations have names that match the fields that they operate on. The 
`max` aggregation is named `time` and its field also needs to be `time`.
<2> The `term` aggregation is named `airline` and its field is also named 
`airline`.
<3> The `avg` aggregation is named `responsetime` and its field is also named 
`responsetime`.

Your {dfeed} can contain multiple aggregations, but only the ones with names 
that match values in the job configuration are fed to the job.


[discrete]
[[aggs-dfeeds]]
==== Nested aggregations in {dfeeds}

{dfeeds-cap} support complex nested aggregations. This example uses the 
`derivative` pipeline aggregation to find the first order derivative of the 
counter `system.network.out.bytes` for each value of the field `beat.name`.

[source,js]
----------------------------------
"aggregations": {
  "beat.name": {
    "terms": {
      "field": "beat.name"
    },
    "aggregations": {
      "buckets": {
        "date_histogram": {
          "field": "@timestamp",
          "fixed_interval": "5m"
        },
        "aggregations": {
          "@timestamp": {
            "max": {
              "field": "@timestamp"
            }
{'fraction_non_alphanumeric': 0.12301552106430155, 'fraction_numerical': 0.0038137472283813747, 'mean_word_length': 3.0099573257467993, 'pattern_counts': {'":': 84, '<': 39, '<?xml version=': 0, '>': 39, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1886618', 'n_tokens_mistral': 3254, 'n_tokens_neox': 3081, 'n_words': 1342}