<commit_before>[[analysis-pattern-analyzer]]
=== Pattern Analyzer

An analyzer of type `pattern` that can flexibly separate text into terms
via a regular expression. Accepts the following settings:

The following are settings that can be set for a `pattern` analyzer
type:

[cols="<,<",options="header",]
|===================================================================
|Setting |Description
|`lowercase` |Should terms be lowercased or not. Defaults to `true`.
|`pattern` |The regular expression pattern, defaults to `\W+`.
|`flags` |The regular expression flags.
|`stopwords` |A list of stopwords to initialize the stop filter with.
Defaults to an 'empty' stopword list Check
<<analysis-stop-analyzer,Stop Analyzer>> for more details.
|===================================================================

*IMPORTANT*: The regular expression should match the *token separators*,
not the tokens themselves.

Flags should be pipe-separated, eg `"CASE_INSENSITIVE|COMMENTS"`. Check
http://download.oracle.com/javase/6/docs/api/java/util/regex/Pattern.html#field_summary[Java
Pattern API] for more details about `flags` options.

[float]
==== Pattern Analyzer Examples

In order to try out these examples, you should delete the `test` index
before running each example:

[source,js]
--------------------------------------------------
    curl -XDELETE localhost:9200/test
--------------------------------------------------

[float]
===== Whitespace tokenizer

[source,js]
--------------------------------------------------
    curl -XPUT 'localhost:9200/test' -d '
    {
        "settings":{
            "analysis": {
                "analyzer": {
                    "whitespace":{
                        "type": "pattern",
                        "pattern":"\\\\s+"
                    }
                }
            }
        }
    }'

    curl 'localhost:9200/test/_analyze?pretty=1&analyzer=whitespace' -d 'foo,bar baz'
    # "foo,bar", "baz"
--------------------------------------------------

[float]
===== Non-word character tokenizer

[source,js]
--------------------------------------------------

    curl -XPUT 'localhost:9200/test' -d '
    {
        "settings":{
            "analysis": {
                "analyzer": {
                    "nonword":{
                        "type": "pattern",
                        "pattern":"[^\\\\w]+"
                    }
                }
            }
        }
    }'

    curl 'localhost:9200/test/_analyze?pretty=1&analyzer=nonword' -d 'foo,bar baz'
    # "foo,bar baz" becomes "foo", "bar", "baz"

    curl 'localhost:9200/test/_analyze?pretty=1&analyzer=nonword' -d 'type_1-type_4'
    # "type_1","type_4"
--------------------------------------------------

[float]
===== CamelCase tokenizer

[source,js]
--------------------------------------------------

    curl -XPUT 'localhost:9200/test?pretty=1' -d '
    {
        "settings":{
            "analysis": {
                "analyzer": {
                    "camel":{
                        "type": "pattern",
                        "pattern":"([^\\\\p{L}\\\\d]+)|(?<=\\\\D)(?=\\\\d)|(?<=\\\\d)(?=\\\\D)|(?<=[\\\\p{L}&&[^\\\\p{Lu}]])(?=\\\\p{Lu})|(?<=\\\\p{Lu})(?=\\\\p{Lu}[\\\\p{L}&&[^\\\\p{Lu}]])"
                    }
                }
            }
        }
    }'

    curl 'localhost:9200/test/_analyze?pretty=1&analyzer=camel' -d '
        MooseX::FTPClass2_beta
    '
    # "moose","x","ftp","class","2","beta"
--------------------------------------------------

The regex above is easier to understand as:

[source,js]
--------------------------------------------------

      ([^\\p{L}\\d]+)                 # swallow non letters and numbers,
    | (?<=\\D)(?=\\d)                 # or non-number followed by number,
    | (?<=\\d)(?=\\D)                 # or number followed by non-number,
    | (?<=[ \\p{L} && [^\\p{Lu}]])    # or lower case
      (?=\\p{Lu})                    #   followed by upper case,
    | (?<=\\p{Lu})                   # or upper case
      (?=\\p{Lu}                     #   followed by upper case
        [\\p{L}&&[^\\p{Lu}]]          #   then lower case
      )
--------------------------------------------------<commit_msg>Docs: Fixed the backslash escaping on the pattern analyzer docs<commit_after>[[analysis-pattern-analyzer]]
=== Pattern Analyzer

An analyzer of type `pattern` that can flexibly separate text into terms
via a regular expression. Accepts the following settings:

The following are settings that can be set for a `pattern` analyzer
type:

[horizontal]
`lowercase`::   Should terms be lowercased or not. Defaults to `true`.
`pattern`::     The regular expression pattern, defaults to `\W+`.
`flags`::       The regular expression flags.
`stopwords`::   A list of stopwords to initialize the stop filter with.
                Defaults to an 'empty' stopword list Check
                <<analysis-stop-analyzer,Stop Analyzer>> for more details.

*IMPORTANT*: The regular expression should match the *token separators*,
not the tokens themselves.

Flags should be pipe-separated, eg `"CASE_INSENSITIVE|COMMENTS"`. Check
http://download.oracle.com/javase/6/docs/api/java/util/regex/Pattern.html#field_summary[Java
Pattern API] for more details about `flags` options.

[float]
==== Pattern Analyzer Examples

In order to try out these examples, you should delete the `test` index
before running each example.

[float]
===== Whitespace tokenizer

[source,js]
--------------------------------------------------
DELETE test

PUT /test
{
  "settings": {
    "analysis": {
      "analyzer": {
        "whitespace": {
          "type": "pattern",
          "pattern": "\\s+"
        }
      }
    }
  }
}

GET /test/_analyze?analyzer=whitespace&text=foo,bar baz

# "foo,bar", "baz"
--------------------------------------------------
// AUTOSENSE

[float]
===== Non-word character tokenizer

[source,js]
--------------------------------------------------
DELETE test

PUT /test
{
  "settings": {
    "analysis": {
      "analyzer": {
        "nonword": {
          "type": "pattern",
          "pattern": "[^\\w]+" <1>
        }
      }
    }
  }
}

GET /test/_analyze?analyzer=nonword&text=foo,bar baz
# "foo,bar baz" becomes "foo", "bar", "baz"

GET /test/_analyze?analyzer=nonword&text=type_1-type_4
# "type_1","type_4"
--------------------------------------------------
// AUTOSENSE


[float]
===== CamelCase tokenizer

[source,js]
--------------------------------------------------
DELETE test

PUT /test?pretty=1
{
  "settings": {
    "analysis": {
      "analyzer": {
        "camel": {
          "type": "pattern",
          "pattern": "([^\\p{L}\\d]+)|(?<=\\D)(?=\\d)|(?<=\\d)(?=\\D)|(?<=[\\p{L}&&[^\\p{Lu}]])(?=\\p{Lu})|(?<=\\p{Lu})(?=\\p{Lu}[\\p{L}&&[^\\p{Lu}]])"
        }
      }
    }
  }
}

GET /test/_analyze?analyzer=camel&text=MooseX::FTPClass2_beta
# "moose","x","ftp","class","2","beta"
--------------------------------------------------
// AUTOSENSE

The regex above is easier to understand as:

[source,js]
--------------------------------------------------

  ([^\p{L}\d]+)                 # swallow non letters and numbers,
| (?<=\D)(?=\d)                 # or non-number followed by number,
| (?<=\d)(?=\D)                 # or number followed by non-number,
| (?<=[ \p{L} && [^\p{Lu}]])    # or lower case
  (?=\p{Lu})                    #   followed by upper case,
| (?<=\p{Lu})                   # or upper case
  (?=\p{Lu}                     #   followed by upper case
    [\p{L}&&[^\p{Lu}]]          #   then lower case
  )
--------------------------------------------------
{'fraction_non_alphanumeric': 0.2917876947451809, 'fraction_numerical': 0.006997623448640085, 'mean_word_length': 1.5791624106230848, 'pattern_counts': {'":': 36, '<': 26, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '825884', 'n_tokens_mistral': 2289, 'n_tokens_neox': 2162, 'n_words': 528}