<commit_before>[[grok]]
=== Grokking grok
Grok is a regular expression dialect that supports reusable aliased expressions. Grok works really well with syslog logs, Apache and other webserver
logs, mysql logs, and generally any log format that is written for humans and
not computer consumption.

Grok sits on top of the https://github.com/kkos/oniguruma/blob/master/doc/RE[Oniguruma] regular expression library, so any regular expressions are
valid in grok. Grok uses this regular expression language to allow naming
existing patterns and combining them into more complex patterns that match your
fields.

[[grok-syntax]]
==== Grok patterns
The {stack} ships with numerous https://github.com/elastic/elasticsearch/blob/master/libs/grok/src/main/resources/patterns/grok-patterns[predefined grok patterns] that simplify working with grok. The syntax for reusing grok patterns
takes one of the following forms:

[%autowidth]
|===
|`%{SYNTAX}` | `%{SYNTAX:ID}` |`%{SYNTAX:ID:TYPE}`
|===

`SYNTAX`::
The name of the pattern that will match your text. For example, `NUMBER` and
`IP` are both patterns that are provided within the default patterns set. The
`NUMBER` pattern matches data like `3.44`, and the `IP` pattern matches data
like `172.16.58.3`. 

`ID`::
The identifier you give to the piece of text being matched. For  example, `3.44`
could be the duration of an event, so you might call it `duration`. The string
`172.16.58.3` might identify the `client` making a request.

`TYPE`::
The data type you want to cast your named field. `int`, `long`, `double`,
`float` and `boolean` are supported types.

For example, let's say you have message data that looks like this:

[source,txt]
----
3.44 172.16.58.3
----

The first value is a number, followed by what appears to be an IP address. You
can match this text by using the following grok expression:

[source,txt]
----
%{NUMBER:duration} %{IP:client}
----

[[grok-patterns]]
==== Use grok patterns in Painless scripts
You can incorporate predefined grok patterns into Painless scripts to extract
data. To test your script, use either the {painless}/painless-execute-api.html#painless-execute-runtime-field-context[field contexts] of the Painless
execute API or create a runtime field that includes the script. Runtime fields
offer greater flexibility and accept multiple documents, but the Painless
execute API is a great option if you don't have write access on a cluster
where you're testing a script.

TIP: If you need help building grok patterns to match your data, use the 
{kibana-ref}/xpack-grokdebugger.html[Grok Debugger] tool in {kib}.

For example, if you're working with Apache log data, you can use the
`%{COMMONAPACHELOG}` syntax, which understands the structure of Apache logs. A
sample document might look like this:

// Note to contributors that the line break in the following example is
// intentional to promote better readability in the output
[source,js]
----<commit_msg>Fix broken link for grok patterns in documentation (#78878)<commit_after>[[grok]]
=== Grokking grok
Grok is a regular expression dialect that supports reusable aliased expressions. Grok works really well with syslog logs, Apache and other webserver
logs, mysql logs, and generally any log format that is written for humans and
not computer consumption.

Grok sits on top of the https://github.com/kkos/oniguruma/blob/master/doc/RE[Oniguruma] regular expression library, so any regular expressions are
valid in grok. Grok uses this regular expression language to allow naming
existing patterns and combining them into more complex patterns that match your
fields.

[[grok-syntax]]
==== Grok patterns
The {stack} ships with numerous https://github.com/elastic/elasticsearch/blob/master/libs/grok/src/main/resources/patterns/legacy/grok-patterns[predefined grok patterns] that simplify working with grok. The syntax for reusing grok patterns
takes one of the following forms:

[%autowidth]
|===
|`%{SYNTAX}` | `%{SYNTAX:ID}` |`%{SYNTAX:ID:TYPE}`
|===

`SYNTAX`::
The name of the pattern that will match your text. For example, `NUMBER` and
`IP` are both patterns that are provided within the default patterns set. The
`NUMBER` pattern matches data like `3.44`, and the `IP` pattern matches data
like `172.16.58.3`. 

`ID`::
The identifier you give to the piece of text being matched. For  example, `3.44`
could be the duration of an event, so you might call it `duration`. The string
`172.16.58.3` might identify the `client` making a request.

`TYPE`::
The data type you want to cast your named field. `int`, `long`, `double`,
`float` and `boolean` are supported types.

For example, let's say you have message data that looks like this:

[source,txt]
----
3.44 172.16.58.3
----

The first value is a number, followed by what appears to be an IP address. You
can match this text by using the following grok expression:

[source,txt]
----
%{NUMBER:duration} %{IP:client}
----

[[grok-ecs]]
==== Migrating to Elastic Common Schema (ECS)

To ease migration to the {ecs-ref}[Elastic Common Schema (ECS)], a new set of 
ECS-compliant patterns is available in addition to the existing patterns. The
new ECS pattern definitions capture event field names that are compliant with
the schema.

The ECS pattern set has all of the pattern definitions from the legacy set, and
is a drop-in replacement. Use the 
{logstash-ref}/plugins-filters-grok.html#plugins-filters-grok-ecs_compatibility[`ecs-compatability`]
setting to switch modes. 

New features and enhancements will be added to the ECS-compliant files. The
legacy patterns may still receive bug fixes which are backwards compatible.

[[grok-patterns]]
==== Use grok patterns in Painless scripts
You can incorporate predefined grok patterns into Painless scripts to extract
data. To test your script, use either the {painless}/painless-execute-api.html#painless-execute-runtime-field-context[field contexts] of the Painless
execute API or create a runtime field that includes the script. Runtime fields
offer greater flexibility and accept multiple documents, but the Painless
execute API is a great option if you don't have write access on a cluster
where you're testing a script.

TIP: If you need help building grok patterns to match your data, use the 
{kibana-ref}/xpack-grokdebugger.html[Grok Debugger] tool in {kib}.

For example, if you're working with Apache log data, you can use the
`%{COMMONAPACHELOG}` syntax, which understands the structure of Apache logs. A
sample document might look like this:

// Note to contributors that the line break in the following example is
// intentional to promote better readability in the output
[source,js]
----
{'fraction_non_alphanumeric': 0.08620430269294418, 'fraction_numerical': 0.010681510455844742, 'mean_word_length': 4.643463497453311, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 6, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3097693', 'n_tokens_mistral': 1926, 'n_tokens_neox': 1848, 'n_words': 904}