<commit_before>[[analysis-word-delimiter-graph-tokenfilter]]
=== Word delimiter graph token filter
++++
<titleabbrev>Word delimiter graph</titleabbrev>
++++

Splits tokens at non-alphanumeric characters. The `word_delimiter_graph` filter
also performs optional token normalization based on a set of rules. By default,
the filter uses the following rules:

* Split tokens at non-alphanumeric characters.
  The filter uses these characters as delimiters.
  For example: `Super-Duper` -> `Super`, `Duper`
* Remove leading or trailing delimiters from each token.
  For example: `XL---42+'Autocoder'` -> `XL`, `42`, `Autocoder`
* Split tokens at letter case transitions.
  For example: `PowerShot` -> `Power`, `Shot`
* Split tokens at letter-number transitions.
  For example: `XL500` -> `XL`, `500`
* Remove the English possessive (`'s`) from the end of each token.
  For example: `Neil's` -> `Neil`

The `word_delimiter_graph` filter uses Lucene's
{lucene-analysis-docs}/miscellaneous/WordDelimiterGraphFilter.html[WordDelimiterGraphFilter].

[TIP]
====
The `word_delimiter_graph` filter was designed to remove punctuation from
complex identifiers, such as product IDs or part numbers. For these use cases,
we recommend using the `word_delimiter_graph` filter with the
<<analysis-keyword-tokenizer,`keyword`>> tokenizer.

Avoid using the `word_delimiter_graph` filter to split hyphenated words, such as
`wi-fi`. Because users often search for these words both with and without
hyphens, we recommend using the
<<analysis-synonym-graph-tokenfilter,`synonym_graph`>> filter instead.
====

[[analysis-word-delimiter-graph-tokenfilter-analyze-ex]]
==== Example

The following <<indices-analyze,analyze API>> request uses the
`word_delimiter_graph` filter to split `Neil's Super-Duper-XL500--42+AutoCoder`
into normalized tokens using the filter's default rules:

[source,console]
----
GET /_analyze
{
  "tokenizer": "whitespace",
  "filter": [ "word_delimiter_graph" ],
  "text": "Neil's Super-Duper-XL500--42+AutoCoder"
}
----

The filter produces the following tokens:

[source,txt]
----
[ Neil, Super, Duper, XL, 500, 42, Auto, Coder ]
----

////
[source,console-result]
----
{
  "tokens" : [
    {
      "token" : "Neil",
      "start_offset" : 0,
      "end_offset" : 4,
      "type" : "word",
      "position" : 0
    },
    {
      "token" : "Super",
      "start_offset" : 7,
      "end_offset" : 12,
      "type" : "word",
      "position" : 1
    },
    {
      "token" : "Duper",
      "start_offset" : 13,
      "end_offset" : 18,
      "type" : "word",
      "position" : 2
    },
    {
      "token" : "XL",
      "start_offset" : 19,
      "end_offset" : 21,
      "type" : "word",
      "position" : 3
    },
    {
      "token" : "500",
      "start_offset" : 21,
      "end_offset" : 24,
      "type" : "word",
      "position" : 4
    },
    {
      "token" : "42",
      "start_offset" : 26,
      "end_offset" : 28,
      "type" : "word",
      "position" : 5
    },
    {
      "token" : "Auto",
      "start_offset" : 29,
      "end_offset" : 33,
      "type" : "word",
      "position" : 6
    },
    {
      "token" : "Coder",
      "start_offset" : 33,
      "end_offset" : 38,
      "type" : "word",
      "position" : 7
    }
  ]
}
----
////

[[analysis-word-delimiter-graph-tokenfilter-analyzer-ex]]
==== Add to an analyzer

The following <<indices-create-index,create index API>> request uses the
`word_delimiter_graph` filter to configure a new
<<analysis-custom-analyzer,custom analyzer>>.

[source,console]
----
PUT /my_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_analyzer": {
          "tokenizer": "whitespace",
          "filter": [ "word_delimiter_graph" ]
        }
      }
    }
  }
}
----

[WARNING]
====
Avoid using the `word_delimiter_graph` filter with tokenizers that remove
punctuation, such as the <<analysis-standard-tokenizer,`standard`>> tokenizer.
This could prevent the `word_delimiter_graph` filter from splitting tokens
correctly. It can also interfere with the filter's configurable parameters, such
as <<word-delimiter-graph-tokenfilter-catenate-all,`catenate_all`>> or
<<word-delimiter-graph-tokenfilter-preserve-original,`preserve_original`>>. We
recommend using the <<analysis-keyword-tokenizer,`keyword`>> or
<<analysis-whitespace-tokenizer,`whitespace`>> tokenizer instead.
====

[[word-delimiter-graph-tokenfilter-configure-parms]]
==== Configurable parameters

[[word-delimiter-graph-tokenfilter-adjust-offsets]]
`adjust_offsets`::
+
--
(Optional, boolean)
If `true`, the filter adjusts the offsets of split or catenated tokens to better
reflect their actual position in the token stream. Defaults to `true`.

[WARNING]
====
Set `adjust_offsets` to `false` if your analyzer uses filters, such as the
<<analysis-trim-tokenfilter,`trim`>> filter, that change the length of tokens
without changing their offsets. Otherwise, the `word_delimiter_graph` filter
could produce tokens with illegal offsets.
====
--

[[word-delimiter-graph-tokenfilter-catenate-all]]
`catenate_all`::
+
--
(Optional, boolean)
If `true`, the filter produces catenated tokens for chains of alphanumeric
characters separated by non-alphabetic delimiters. For example:
`super-duper-xl-500` -> [**`superduperxl500`**, `super`, `duper`, `xl`, `500` ].
Defaults to `false`.

[WARNING]
====
Setting this parameter to `true` produces multi-position tokens, which are not
supported by indexing.

If this parameter is `true`, avoid using this filter in an index analyzer or
use the <<analysis-flatten-graph-tokenfilter,`flatten_graph`>> filter after
this filter to make the token stream suitable for indexing.

When used for search analysis, catenated tokens can cause problems for the
<<query-dsl-match-query-phrase,`match_phrase`>> query and other queries that
rely on token position for matching. Avoid setting this parameter to `true` if
you plan to use these queries.
====
--

[[word-delimiter-graph-tokenfilter-catenate-numbers]]
`catenate_numbers`::
+
--
(Optional, boolean)
If `true`, the filter produces catenated tokens for chains of numeric characters
separated by non-alphabetic delimiters. For example: `01-02-03` ->
[**`010203`**, `01`, `02`, `03` ]. Defaults to `false`.

[WARNING]
====
Setting this parameter to `true` produces multi-position tokens, which are not
supported by indexing.

If this parameter is `true`, avoid using this filter in an index analyzer or
use the <<analysis-flatten-graph-tokenfilter,`flatten_graph`>> filter after
this filter to make the token stream suitable for indexing.

When used for search analysis, catenated tokens can cause problems for the
<<query-dsl-match-query-phrase,`match_phrase`>> query and other queries that
rely on token position for matching. Avoid setting this parameter to `true` if
you plan to use these queries.
====
--

[[word-delimiter-graph-tokenfilter-catenate-words]]
`catenate_words`::
+
--
(Optional, boolean)
If `true`, the filter produces catenated tokens for chains of alphabetical
characters separated by non-alphabetic delimiters. For example: `super-duper-xl`
-> [**`superduperxl`**, `super`, `duper`, `xl`]. Defaults to `false`.

[WARNING]
====
Setting this parameter to `true` produces multi-position tokens, which are not
supported by indexing.

If this parameter is `true`, avoid using this filter in an index analyzer or
use the <<analysis-flatten-graph-tokenfilter,`flatten_graph`>> filter after
this filter to make the token stream suitable for indexing.

When used for search analysis, catenated tokens can cause problems for the
<<query-dsl-match-query-phrase,`match_phrase`>> query and other queries that
rely on token position for matching. Avoid setting this parameter to `true` if
you plan to use these queries.
====
--

`generate_number_parts`::
(Optional, boolean)
If `true`, the filter includes tokens consisting of only numeric characters in
the output. If `false`, the filter excludes these tokens from the output.
Defaults to `true`.

`generate_word_parts`::
(Optional, boolean)
If `true`, the filter includes tokens consisting of only alphabetical characters
in the output. If `false`, the filter excludes these tokens from the output.
Defaults to `true`.

[[word-delimiter-graph-tokenfilter-preserve-original]]
`preserve_original`::
+
--
(Optional, boolean)
If `true`, the filter includes the original version of any split tokens in the
output. This original version includes non-alphanumeric delimiters. For example:
`super-duper-xl-500` -> [**`super-duper-xl-500`**, `super`, `duper`, `xl`, `500`
]. Defaults to `false`.

[WARNING]
====
Setting this parameter to `true` produces multi-position tokens, which are not
supported by indexing.

If this parameter is `true`, avoid using this filter in an index analyzer or
use the <<analysis-flatten-graph-tokenfilter,`flatten_graph`>> filter after
this filter to make the token stream suitable for indexing.
====
--

`protected_words`::
(Optional, array of strings)
Array of tokens the filter won't split.

`protected_words_path`::
+
--
(Optional, string)
Path to a file that contains a list of tokens the filter won't split.

This path must be absolute or relative to the `config` location, and the file
must be UTF-8 encoded. Each token in the file must be separated by a line
break.
--

`split_on_case_change`::
(Optional, boolean)
If `true`, the filter splits tokens at letter case transitions. For example:
`camelCase` -> [ `camel`, `Case`]. Defaults to `true`.

`split_on_numerics`::
(Optional, boolean)
If `true`, the filter splits tokens at letter-number transitions. For example:
`j2se` -> [ `j`, `2`, `se` ]. Defaults to `true`.

`stem_english_possessive`::
(Optional, boolean)
If `true`, the filter removes the English possessive (`'s`) from the end of each
token. For example: `O'Neil's` -> `[ `O`, `Neil` ]. Defaults to `true`.

`type_table`::
+
--
(Optional, array of strings)
Array of custom type mappings for characters. This allows you to map
non-alphanumeric characters as numeric or alphanumeric to avoid splitting on
those characters.

For example, the following array maps the plus (`+`) and hyphen (`-`) characters
as alphanumeric, which means they won't be treated as delimiters:

`["+ => ALPHA", "- => ALPHA"]`

Supported types include:

* `ALPHA` (Alphabetical)
* `ALPHANUM` (Alphanumeric)
* `DIGIT` (Numeric)
* `LOWER` (Lowercase alphabetical)
* `SUBWORD_DELIM` (Non-alphanumeric delimiter)
* `UPPER` (Uppercase alphabetical)
--

`type_table_path`::
+
--
(Optional, string)
Path to a file that contains custom type mappings for characters. This allows
you to map non-alphanumeric characters as numeric or alphanumeric to avoid
splitting on those characters.

For example, the contents of this file may contain the following:

[source,txt]
----
# Map the $, %, '.', and ',' characters to DIGIT
# This might be useful for financial data.
$ => DIGIT
% => DIGIT
. => DIGIT
\\u002C => DIGIT

# in some cases you might not want to split on ZWJ
# this also tests the case where we need a bigger byte[]
# see http://en.wikipedia.org/wiki/Zero-width_joiner
\\u200D => ALPHANUM
----

Supported types include:

* `ALPHA` (Alphabetical)
* `ALPHANUM` (Alphanumeric)
* `DIGIT` (Numeric)
* `LOWER` (Lowercase alphabetical)
* `SUBWORD_DELIM` (Non-alphanumeric delimiter)
* `UPPER` (Uppercase alphabetical)

This file path must be absolute or relative to the `config` location, and the
file must be UTF-8 encoded. Each mapping in the file must be separated by a line
break.
--

[[analysis-word-delimiter-graph-tokenfilter-customize]]
==== Customize

To customize the `word_delimiter_graph` filter, duplicate it to create the basis
for a new custom token filter. You can modify the filter using its configurable
parameters.

For example, the following request creates a `word_delimiter_graph`
filter that uses the following rules:

* Split tokens at non-alphanumeric characters, _except_ the hyphen (`-`)
  character.
* Remove leading or trailing delimiters from each token.
* Do _not_ split tokens at letter case transitions.
* Do _not_ split tokens at letter-number transitions.
* Remove the English possessive (`'s`) from the end of each token.

[source,console]
----
PUT /my_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_analyzer": {
          "tokenizer": "whitespace",
          "filter": [ "my_custom_word_delimiter_graph_filter" ]
        }
      },
      "filter": {
        "my_custom_word_delimiter_graph_filter": {
          "type": "word_delimiter_graph",
          "type_table": [ "- => ALPHA" ],
          "split_on_case_change": false,
          "split_on_numerics": false,<commit_msg>[DOCS] Use keyword tokenizer in word delimiter graph examples (#53384)<commit_after>[[analysis-word-delimiter-graph-tokenfilter]]
=== Word delimiter graph token filter
++++
<titleabbrev>Word delimiter graph</titleabbrev>
++++

Splits tokens at non-alphanumeric characters. The `word_delimiter_graph` filter
also performs optional token normalization based on a set of rules. By default,
the filter uses the following rules:

* Split tokens at non-alphanumeric characters.
  The filter uses these characters as delimiters.
  For example: `Super-Duper` -> `Super`, `Duper`
* Remove leading or trailing delimiters from each token.
  For example: `XL---42+'Autocoder'` -> `XL`, `42`, `Autocoder`
* Split tokens at letter case transitions.
  For example: `PowerShot` -> `Power`, `Shot`
* Split tokens at letter-number transitions.
  For example: `XL500` -> `XL`, `500`
* Remove the English possessive (`'s`) from the end of each token.
  For example: `Neil's` -> `Neil`

The `word_delimiter_graph` filter uses Lucene's
{lucene-analysis-docs}/miscellaneous/WordDelimiterGraphFilter.html[WordDelimiterGraphFilter].

[TIP]
====
The `word_delimiter_graph` filter was designed to remove punctuation from
complex identifiers, such as product IDs or part numbers. For these use cases,
we recommend using the `word_delimiter_graph` filter with the
<<analysis-keyword-tokenizer,`keyword`>> tokenizer.

Avoid using the `word_delimiter_graph` filter to split hyphenated words, such as
`wi-fi`. Because users often search for these words both with and without
hyphens, we recommend using the
<<analysis-synonym-graph-tokenfilter,`synonym_graph`>> filter instead.
====

[[analysis-word-delimiter-graph-tokenfilter-analyze-ex]]
==== Example

The following <<indices-analyze,analyze API>> request uses the
`word_delimiter_graph` filter to split `Neil's-Super-Duper-XL500--42+AutoCoder`
into normalized tokens using the filter's default rules:

[source,console]
----
GET /_analyze
{
  "tokenizer": "keyword",
  "filter": [ "word_delimiter_graph" ],
  "text": "Neil's-Super-Duper-XL500--42+AutoCoder"
}
----

The filter produces the following tokens:

[source,txt]
----
[ Neil, Super, Duper, XL, 500, 42, Auto, Coder ]
----

////
[source,console-result]
----
{
  "tokens": [
    {
      "token": "Neil",
      "start_offset": 0,
      "end_offset": 4,
      "type": "word",
      "position": 0
    },
    {
      "token": "Super",
      "start_offset": 7,
      "end_offset": 12,
      "type": "word",
      "position": 1
    },
    {
      "token": "Duper",
      "start_offset": 13,
      "end_offset": 18,
      "type": "word",
      "position": 2
    },
    {
      "token": "XL",
      "start_offset": 19,
      "end_offset": 21,
      "type": "word",
      "position": 3
    },
    {
      "token": "500",
      "start_offset": 21,
      "end_offset": 24,
      "type": "word",
      "position": 4
    },
    {
      "token": "42",
      "start_offset": 26,
      "end_offset": 28,
      "type": "word",
      "position": 5
    },
    {
      "token": "Auto",
      "start_offset": 29,
      "end_offset": 33,
      "type": "word",
      "position": 6
    },
    {
      "token": "Coder",
      "start_offset": 33,
      "end_offset": 38,
      "type": "word",
      "position": 7
    }
  ]
}
----
////

[[analysis-word-delimiter-graph-tokenfilter-analyzer-ex]]
==== Add to an analyzer

The following <<indices-create-index,create index API>> request uses the
`word_delimiter_graph` filter to configure a new
<<analysis-custom-analyzer,custom analyzer>>.

[source,console]
----
PUT /my_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_analyzer": {
          "tokenizer": "keyword",
          "filter": [ "word_delimiter_graph" ]
        }
      }
    }
  }
}
----

[WARNING]
====
Avoid using the `word_delimiter_graph` filter with tokenizers that remove
punctuation, such as the <<analysis-standard-tokenizer,`standard`>> tokenizer.
This could prevent the `word_delimiter_graph` filter from splitting tokens
correctly. It can also interfere with the filter's configurable parameters, such
as <<word-delimiter-graph-tokenfilter-catenate-all,`catenate_all`>> or
<<word-delimiter-graph-tokenfilter-preserve-original,`preserve_original`>>. We
recommend using the <<analysis-keyword-tokenizer,`keyword`>> or
<<analysis-whitespace-tokenizer,`whitespace`>> tokenizer instead.
====

[[word-delimiter-graph-tokenfilter-configure-parms]]
==== Configurable parameters

[[word-delimiter-graph-tokenfilter-adjust-offsets]]
`adjust_offsets`::
+
--
(Optional, boolean)
If `true`, the filter adjusts the offsets of split or catenated tokens to better
reflect their actual position in the token stream. Defaults to `true`.

[WARNING]
====
Set `adjust_offsets` to `false` if your analyzer uses filters, such as the
<<analysis-trim-tokenfilter,`trim`>> filter, that change the length of tokens
without changing their offsets. Otherwise, the `word_delimiter_graph` filter
could produce tokens with illegal offsets.
====
--

[[word-delimiter-graph-tokenfilter-catenate-all]]
`catenate_all`::
+
--
(Optional, boolean)
If `true`, the filter produces catenated tokens for chains of alphanumeric
characters separated by non-alphabetic delimiters. For example:
`super-duper-xl-500` -> [ **`superduperxl500`**, `super`, `duper`, `xl`, `500` ].
Defaults to `false`.

[WARNING]
====
Setting this parameter to `true` produces multi-position tokens, which are not
supported by indexing.

If this parameter is `true`, avoid using this filter in an index analyzer or
use the <<analysis-flatten-graph-tokenfilter,`flatten_graph`>> filter after
this filter to make the token stream suitable for indexing.

When used for search analysis, catenated tokens can cause problems for the
<<query-dsl-match-query-phrase,`match_phrase`>> query and other queries that
rely on token position for matching. Avoid setting this parameter to `true` if
you plan to use these queries.
====
--

[[word-delimiter-graph-tokenfilter-catenate-numbers]]
`catenate_numbers`::
+
--
(Optional, boolean)
If `true`, the filter produces catenated tokens for chains of numeric characters
separated by non-alphabetic delimiters. For example: `01-02-03` ->
[ **`010203`**, `01`, `02`, `03` ]. Defaults to `false`.

[WARNING]
====
Setting this parameter to `true` produces multi-position tokens, which are not
supported by indexing.

If this parameter is `true`, avoid using this filter in an index analyzer or
use the <<analysis-flatten-graph-tokenfilter,`flatten_graph`>> filter after
this filter to make the token stream suitable for indexing.

When used for search analysis, catenated tokens can cause problems for the
<<query-dsl-match-query-phrase,`match_phrase`>> query and other queries that
rely on token position for matching. Avoid setting this parameter to `true` if
you plan to use these queries.
====
--

[[word-delimiter-graph-tokenfilter-catenate-words]]
`catenate_words`::
+
--
(Optional, boolean)
If `true`, the filter produces catenated tokens for chains of alphabetical
characters separated by non-alphabetic delimiters. For example: `super-duper-xl`
-> [ **`superduperxl`**, `super`, `duper`, `xl` ]. Defaults to `false`.

[WARNING]
====
Setting this parameter to `true` produces multi-position tokens, which are not
supported by indexing.

If this parameter is `true`, avoid using this filter in an index analyzer or
use the <<analysis-flatten-graph-tokenfilter,`flatten_graph`>> filter after
this filter to make the token stream suitable for indexing.

When used for search analysis, catenated tokens can cause problems for the
<<query-dsl-match-query-phrase,`match_phrase`>> query and other queries that
rely on token position for matching. Avoid setting this parameter to `true` if
you plan to use these queries.
====
--

`generate_number_parts`::
(Optional, boolean)
If `true`, the filter includes tokens consisting of only numeric characters in
the output. If `false`, the filter excludes these tokens from the output.
Defaults to `true`.

`generate_word_parts`::
(Optional, boolean)
If `true`, the filter includes tokens consisting of only alphabetical characters
in the output. If `false`, the filter excludes these tokens from the output.
Defaults to `true`.

[[word-delimiter-graph-tokenfilter-preserve-original]]
`preserve_original`::
+
--
(Optional, boolean)
If `true`, the filter includes the original version of any split tokens in the
output. This original version includes non-alphanumeric delimiters. For example:
`super-duper-xl-500` -> [ **`super-duper-xl-500`**, `super`, `duper`, `xl`,
`500` ]. Defaults to `false`.

[WARNING]
====
Setting this parameter to `true` produces multi-position tokens, which are not
supported by indexing.

If this parameter is `true`, avoid using this filter in an index analyzer or
use the <<analysis-flatten-graph-tokenfilter,`flatten_graph`>> filter after
this filter to make the token stream suitable for indexing.
====
--

`protected_words`::
(Optional, array of strings)
Array of tokens the filter won't split.

`protected_words_path`::
+
--
(Optional, string)
Path to a file that contains a list of tokens the filter won't split.

This path must be absolute or relative to the `config` location, and the file
must be UTF-8 encoded. Each token in the file must be separated by a line
break.
--

`split_on_case_change`::
(Optional, boolean)
If `true`, the filter splits tokens at letter case transitions. For example:
`camelCase` -> [ `camel`, `Case` ]. Defaults to `true`.

`split_on_numerics`::
(Optional, boolean)
If `true`, the filter splits tokens at letter-number transitions. For example:
`j2se` -> [ `j`, `2`, `se` ]. Defaults to `true`.

`stem_english_possessive`::
(Optional, boolean)
If `true`, the filter removes the English possessive (`'s`) from the end of each
token. For example: `O'Neil's` -> [ `O`, `Neil` ]. Defaults to `true`.

`type_table`::
+
--
(Optional, array of strings)
Array of custom type mappings for characters. This allows you to map
non-alphanumeric characters as numeric or alphanumeric to avoid splitting on
those characters.

For example, the following array maps the plus (`+`) and hyphen (`-`) characters
as alphanumeric, which means they won't be treated as delimiters:

`[ "+ => ALPHA", "- => ALPHA" ]`

Supported types include:

* `ALPHA` (Alphabetical)
* `ALPHANUM` (Alphanumeric)
* `DIGIT` (Numeric)
* `LOWER` (Lowercase alphabetical)
* `SUBWORD_DELIM` (Non-alphanumeric delimiter)
* `UPPER` (Uppercase alphabetical)
--

`type_table_path`::
+
--
(Optional, string)
Path to a file that contains custom type mappings for characters. This allows
you to map non-alphanumeric characters as numeric or alphanumeric to avoid
splitting on those characters.

For example, the contents of this file may contain the following:

[source,txt]
----
# Map the $, %, '.', and ',' characters to DIGIT
# This might be useful for financial data.
$ => DIGIT
% => DIGIT
. => DIGIT
\\u002C => DIGIT

# in some cases you might not want to split on ZWJ
# this also tests the case where we need a bigger byte[]
# see http://en.wikipedia.org/wiki/Zero-width_joiner
\\u200D => ALPHANUM
----

Supported types include:

* `ALPHA` (Alphabetical)
* `ALPHANUM` (Alphanumeric)
* `DIGIT` (Numeric)
* `LOWER` (Lowercase alphabetical)
* `SUBWORD_DELIM` (Non-alphanumeric delimiter)
* `UPPER` (Uppercase alphabetical)

This file path must be absolute or relative to the `config` location, and the
file must be UTF-8 encoded. Each mapping in the file must be separated by a line
break.
--

[[analysis-word-delimiter-graph-tokenfilter-customize]]
==== Customize

To customize the `word_delimiter_graph` filter, duplicate it to create the basis
for a new custom token filter. You can modify the filter using its configurable
parameters.

For example, the following request creates a `word_delimiter_graph`
filter that uses the following rules:

* Split tokens at non-alphanumeric characters, _except_ the hyphen (`-`)
  character.
* Remove leading or trailing delimiters from each token.
* Do _not_ split tokens at letter case transitions.
* Do _not_ split tokens at letter-number transitions.
* Remove the English possessive (`'s`) from the end of each token.

[source,console]
----
PUT /my_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_analyzer": {
          "tokenizer": "keyword",
          "filter": [ "my_custom_word_delimiter_graph_filter" ]
        }
      },
      "filter": {
        "my_custom_word_delimiter_graph_filter": {
          "type": "word_delimiter_graph",
          "type_table": [ "- => ALPHA" ],
          "split_on_case_change": false,
          "split_on_numerics": false,
{'fraction_non_alphanumeric': 0.1245253164556962, 'fraction_numerical': 0.009137658227848101, 'mean_word_length': 4.102119071644803, 'pattern_counts': {'":': 83, '<': 79, '<?xml version=': 0, '>': 119, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2146707', 'n_tokens_mistral': 8020, 'n_tokens_neox': 7669, 'n_words': 2734}