<commit_before>[role="xpack"]
[testenv="basic"]
[[eql-search]]
== Run an EQL search

experimental::[]

To start using EQL in {es}, first ensure your event data meets
<<eql-requirements,EQL requirements>>. You can then use the <<eql-search-api,EQL
search API>> to search event data stored in one or more {es} indices.

.*Example*
[%collapsible]
====
To get started, ingest or add the data to an {es} index.

The following <<docs-bulk,bulk API>> request adds some example log data to the
`sec_logs` index. This log data follows the {ecs-ref}[Elastic Common Schema
(ECS)].

[source,console]
----
PUT /sec_logs/_bulk?refresh
{"index":{"_index" : "sec_logs", "_id" : "1"}}
{ "@timestamp": "2020-12-06T11:04:05.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process" }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "sec_logs", "_id" : "2"}}
{ "@timestamp": "2020-12-06T11:04:07.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "file" }, "file": { "accessed": "2020-12-07T11:07:08.000Z", "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe", "type": "file", "size": 16384 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "sec_logs", "_id" : "3"}}
{ "@timestamp": "2020-12-07T11:06:07.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process" }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "sec_logs", "_id" : "4"}}
{ "@timestamp": "2020-12-07T11:07:08.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "file" }, "file": { "accessed": "2020-12-07T11:07:08.000Z", "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe", "type": "file", "size": 16384 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "sec_logs", "_id" : "5"}}
{ "@timestamp": "2020-12-07T11:07:09.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process" }, "process": { "name": "regsvr32.exe", "path": "C:\\Windows\\System32\\regsvr32.exe" } }
----
// TESTSETUP

[TIP]
=====
You also can set up {beats-ref}/getting-started.html[{beats}], such as
{auditbeat-ref}/auditbeat-getting-started.html[{auditbeat}] or
{winlogbeat-ref}/winlogbeat-getting-started.html[{winlogbeat}], to automatically
send and index your event data in {es}. See
{beats-ref}/getting-started.html[Getting started with {beats}].
=====

You can now use the EQL search API to search this index using an EQL query.

The following request searches the `sec_logs` index using the EQL query
specified in the `query` parameter. The EQL query matches events with an
`event.category` of `process` that have a `process.name` of `cmd.exe`.

[source,console]
----
GET /sec_logs/_eql/search
{
  "query": """
    process where process.name == "cmd.exe"
  """
}
----

Because the `sec_log` index follows the ECS, you don't need to specify the
required <<eql-required-fields,event category or timestamp>> fields. The request
uses the `event.category` and `@timestamp` fields by default.

The API returns the following response containing the matching events. Events
in the response are sorted by timestamp, converted to milliseconds since the
https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order.

[source,console-result]
----
{
  "took": 60,
  "timed_out": false,
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "events": [
      {
        "_index": "sec_logs",
        "_id": "1",
        "_score": null,
        "_source": {
          "@timestamp": "2020-12-06T11:04:05.000Z",
          "agent": {
            "id": "8a4f500d"
          },
          "event": {
            "category": "process"
          },
          "process": {
            "name": "cmd.exe",
            "path": "C:\\Windows\\System32\\cmd.exe"
          }
        },
        "sort": [
          1607252645000
        ]
      },
      {
        "_index": "sec_logs",
        "_id": "3",
        "_score": null,
        "_source": {
          "@timestamp": "2020-12-07T11:06:07.000Z",
          "agent": {
            "id": "8a4f500d"
          },
          "event": {
            "category": "process"
          },
          "process": {
            "name": "cmd.exe",
            "path": "C:\\Windows\\System32\\cmd.exe"
          }
        },
        "sort": [
          1607339167000
        ]
      }
    ]
  }
}
----
// TESTRESPONSE[s/"took": 60/"took": $body.took/]
====

[discrete]
[[eql-search-specify-event-category-field]]
=== Specify an event category field

The EQL search API uses `event.category` as the required
<<eql-required-fields,event category field>> by default. You can use the
`event_category_field` parameter to specify another event category field.

.*Example*
[%collapsible]
====
The following request specifies `file.type` as the event category
field.

[source,console]
----
GET /sec_logs/_eql/search
{
   "event_category_field": "file.type",
  "query": """
    file where agent.id == "8a4f500d"
  """
}
----<commit_msg>[DOCS] EQL: Add sequence example to tutorial (#56965)<commit_after>[role="xpack"]
[testenv="basic"]
[[eql-search]]
== Run an EQL search

experimental::[]

To start using EQL in {es}, first ensure your event data meets
<<eql-requirements,EQL requirements>>. You can then use the <<eql-search-api,EQL
search API>> to search event data stored in one or more {es} indices.

.*Example*
[%collapsible]
====
To get started, ingest or add the data to an {es} index.

The following <<docs-bulk,bulk API>> request adds some example log data to the
`sec_logs` index. This log data follows the {ecs-ref}[Elastic Common Schema
(ECS)].

[source,console]
----
PUT /sec_logs/_bulk?refresh
{"index":{"_index" : "sec_logs", "_id" : "1"}}
{ "@timestamp": "2020-12-06T11:04:05.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process" }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "sec_logs", "_id" : "2"}}
{ "@timestamp": "2020-12-06T11:04:07.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "file" }, "file": { "accessed": "2020-12-07T11:07:08.000Z", "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe", "type": "file", "size": 16384 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "sec_logs", "_id" : "3"}}
{ "@timestamp": "2020-12-07T11:06:07.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process" }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "sec_logs", "_id" : "4"}}
{ "@timestamp": "2020-12-07T11:07:08.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "file" }, "file": { "accessed": "2020-12-07T11:07:08.000Z", "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe", "type": "file", "size": 16384 }, "process": { "name": "cmd.exe", "path": "C:\\Windows\\System32\\cmd.exe" } }
{"index":{"_index" : "sec_logs", "_id" : "5"}}
{ "@timestamp": "2020-12-07T11:07:09.000Z", "agent": { "id": "8a4f500d" }, "event": { "category": "process" }, "process": { "name": "regsvr32.exe", "path": "C:\\Windows\\System32\\regsvr32.exe" } }
----
// TESTSETUP

[TIP]
=====
You also can set up {beats-ref}/getting-started.html[{beats}], such as
{auditbeat-ref}/auditbeat-getting-started.html[{auditbeat}] or
{winlogbeat-ref}/winlogbeat-getting-started.html[{winlogbeat}], to automatically
send and index your event data in {es}. See
{beats-ref}/getting-started.html[Getting started with {beats}].
=====

You can now use the EQL search API to search this index using an EQL query.

The following request searches the `sec_logs` index using the EQL query
specified in the `query` parameter. The EQL query matches events with an
`event.category` of `process` that have a `process.name` of `cmd.exe`.

[source,console]
----
GET /sec_logs/_eql/search
{
  "query": """
    process where process.name == "cmd.exe"
  """
}
----

Because the `sec_log` index follows the ECS, you don't need to specify the
required <<eql-required-fields,event category or timestamp>> fields. The request
uses the `event.category` and `@timestamp` fields by default.

The API returns the following response containing the matching events. Events
in the response are sorted by timestamp, converted to milliseconds since the
https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order.

[source,console-result]
----
{
  "took": 60,
  "timed_out": false,
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "events": [
      {
        "_index": "sec_logs",
        "_id": "1",
        "_score": null,
        "_source": {
          "@timestamp": "2020-12-06T11:04:05.000Z",
          "agent": {
            "id": "8a4f500d"
          },
          "event": {
            "category": "process"
          },
          "process": {
            "name": "cmd.exe",
            "path": "C:\\Windows\\System32\\cmd.exe"
          }
        },
        "sort": [
          1607252645000
        ]
      },
      {
        "_index": "sec_logs",
        "_id": "3",
        "_score": null,
        "_source": {
          "@timestamp": "2020-12-07T11:06:07.000Z",
          "agent": {
            "id": "8a4f500d"
          },
          "event": {
            "category": "process"
          },
          "process": {
            "name": "cmd.exe",
            "path": "C:\\Windows\\System32\\cmd.exe"
          }
        },
        "sort": [
          1607339167000
        ]
      }
    ]
  }
}
----
// TESTRESPONSE[s/"took": 60/"took": $body.took/]
====

[discrete]
[[eql-search-sequence]]
=== Search for a sequence of events

Many query languages allow you to match single events. However, EQL's
<<eql-sequences,sequence syntax>> lets you match an ordered series of events.

.*Example*
[%collapsible]
====
The following EQL search request matches a sequence that:

. Starts with an event with:
+
--
* An `event.category` of `file`
* A `file.name` of `cmd.exe`
--
. Followed by an event with:
+
--
* An `event.category` of `process`
* A `process.name` that contains the substring `regsvr32`
--

[source,console]
----
GET /sec_logs/_eql/search
{
  "query": """
    sequence
      [ file where file.name == "cmd.exe" ]
      [ process where stringContains(process.name, "regsvr32") ]
  """
}
----

The API returns the following response. Matching events in
the `hits.sequences.events` property are sorted by
<<eql-search-api-timestamp-field,timestamp>>, converted to milliseconds since
the https://en.wikipedia.org/wiki/Unix_time[Unix epoch], in ascending order.

[source,console-result]
----
{
  "took": 60,
  "timed_out": false,
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "sequences": [
      {
        "events": [
          {
            "_index": "sec_logs",
            "_id": "4",
            "_score": null,
            "_source": {
              "@timestamp": "2020-12-07T11:07:08.000Z",
              "agent": {
                "id": "8a4f500d"
              },
              "event": {
                "category": "file"
              },
              "file": {
                "accessed": "2020-12-07T11:07:08.000Z",
                "name": "cmd.exe",
                "path": "C:\\Windows\\System32\\cmd.exe",
                "type": "file",
                "size": 16384
              },
              "process": {
                "name": "cmd.exe",
                "path": "C:\\Windows\\System32\\cmd.exe"
              }
            },
            "fields": {
              "@timestamp": [
                "1607339228000"
              ]
            },
            "sort": [
              1607339228000
            ]
          },
          {
            "_index": "sec_logs",
            "_id": "5",
            "_score": null,
            "_source": {
              "@timestamp": "2020-12-07T11:07:09.000Z",
              "agent": {
                "id": "8a4f500d"
              },
              "event": {
                "category": "process"
              },
              "process": {
                "name": "regsvr32.exe",
                "path": "C:\\Windows\\System32\\regsvr32.exe"
              }
            },
            "fields": {
              "@timestamp": [
                "1607339229000"
              ]
            },
            "sort": [
              1607339229000
            ]
          }
        ]
      }
    ]
  }
}
----
// TESTRESPONSE[s/"took": 60/"took": $body.took/]

You can further constrain matching event sequences using the `by` keyword.

The following EQL search request adds `by agent.id` to each event item. This
ensures events matching the sequence share the same `agent.id` field value.

[source,console]
----
GET /sec_logs/_eql/search
{
  "query": """
    sequence
      [ file where file.name == "cmd.exe" ] by agent.id
      [ process where stringContains(process.name, "regsvr32") ] by agent.id
  """
}
----

Because the `agent.id` field is shared across all events in the sequence, it
can be included using `sequence by`. The following query is equivalent to the
prior one.

[source,console]
----
GET /sec_logs/_eql/search
{
  "query": """
    sequence  by agent.id
      [ file where file.name == "cmd.exe" ]
      [ process where stringContains(process.name, "regsvr32") ]
  """
}
----

The API returns the following response. The `hits.sequences.join_keys` property
contains the shared `agent.id` value for each matching event.

[source,console-result]
----
{
  "took": 60,
  "timed_out": false,
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "sequences": [
      {
        "join_keys": [
          "<KEY>"
        ],
        "events": [
          {
            "_index": "sec_logs",
            "_id": "4",
            "_score": null,
            "_source": {
              "@timestamp": "2020-12-07T11:07:08.000Z",
              "agent": {
                "id": "8a4f500d"
              },
              "event": {
                "category": "file"
              },
              "file": {
                "accessed": "2020-12-07T11:07:08.000Z",
                "name": "cmd.exe",
                "path": "C:\\Windows\\System32\\cmd.exe",
                "type": "file",
                "size": 16384
              },
              "process": {
                "name": "cmd.exe",
                "path": "C:\\Windows\\System32\\cmd.exe"
              }
            },
            "fields": {
              "@timestamp": [
                "1607339228000"
              ]
            },
            "sort": [
              1607339228000
            ]
          },
          {
            "_index": "sec_logs",
            "_id": "5",
            "_score": null,
            "_source": {
              "@timestamp": "2020-12-07T11:07:09.000Z",
              "agent": {
                "id": "8a4f500d"
              },
              "event": {
                "category": "process"
              },
              "process": {
                "name": "regsvr32.exe",
                "path": "C:\\Windows\\System32\\regsvr32.exe"
              }
            },
            "fields": {
              "@timestamp": [
                "1607339229000"
              ]
            },
            "sort": [
              1607339229000
            ]
          }
        ]
      }
    ]
  }
}
----
// TESTRESPONSE[s/"took": 60/"took": $body.took/]
====

[discrete]
[[eql-search-specify-event-category-field]]
=== Specify an event category field

The EQL search API uses `event.category` as the required
<<eql-required-fields,event category field>> by default. You can use the
`event_category_field` parameter to specify another event category field.

.*Example*
[%collapsible]
====
The following request specifies `file.type` as the event category
field.

[source,console]
----
GET /sec_logs/_eql/search
{
  "event_category_field": "file.type",
  "query": """
    file where agent.id == "8a4f500d"
  """
}
----
{'fraction_non_alphanumeric': 0.18750781543078654, 'fraction_numerical': 0.05058146805051895, 'mean_word_length': 2.182451253481894, 'pattern_counts': {'":': 286, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6120072', 'n_tokens_mistral': 5915, 'n_tokens_neox': 5298, 'n_words': 1270}