<commit_before>[[token-count]]
=== Token count datatype

A field of type `token_count` is really an <<number,`integer`>> field which
accepts string values, analyzes them, then indexes the number of tokens in the
string.

For instance:

[source,js]
--------------------------------------------------
PUT my_index
{
  "mappings": {
    "_doc": {
      "properties": {
        "name": { <1>
          "type": "text",
          "fields": {
            "length": { <2>
              "type":     "token_count",
              "analyzer": "standard"
            }
          }
        }
      }
    }
  }
}

PUT my_index/_doc/1
{ "name": "<NAME>" }

PUT my_index/_doc/2
{ "name": "<NAME>" }

GET my_index/_search
{
  "query": {
    "term": {
      "name.length": 3 <3>
    }
  }
}
--------------------------------------------------
// CONSOLE
<1> The `name` field is an analyzed string field which uses the default `standard` analyzer.
<2> The `name.length` field is a `token_count` <<multi-fields,multi-field>> which will index the number of tokens in the `name` field.
<3> This query matches only the document containing `<NAME>`, as it contains three tokens.


[[token-count-params]]
==== Parameters for `token_count` fields

The following parameters are accepted by `token_count` fields:

[horizontal]

<<analyzer,`analyzer`>>::

    The <<analysis,analyzer>> which should be used to analyze the string
    value. Required. For best performance, use an analyzer without token
    filters.

`enable_position_increments`:: 

Indicates if position increments should be counted. 
Set to `false` if you don't want to count tokens removed by analyzer filters (like <<analysis-stop-tokenfilter,`stop`>>). 
Defaults to `true`.

<<mapping-boost,`boost`>>::

    Mapping field-level query time boosting. Accepts a floating point number, defaults
    to `1.0`.

<<doc-values,`doc_values`>>::

    Should the field be stored on disk in a column-stride fashion, so that it
    can later be used for sorting, aggregations, or scripting? Accepts `true`
    (default) or `false`.

<<mapping-index,`index`>>::

    Should the field be searchable? Accepts `not_analyzed` (default) and `no`.

<<null-value,`null_value`>>::

    Accepts a numeric value of the same `type` as the field which is<commit_msg>Remove reference to `not_analyzed`.<commit_after>[[token-count]]
=== Token count datatype

A field of type `token_count` is really an <<number,`integer`>> field which
accepts string values, analyzes them, then indexes the number of tokens in the
string.

For instance:

[source,js]
--------------------------------------------------
PUT my_index
{
  "mappings": {
    "_doc": {
      "properties": {
        "name": { <1>
          "type": "text",
          "fields": {
            "length": { <2>
              "type":     "token_count",
              "analyzer": "standard"
            }
          }
        }
      }
    }
  }
}

PUT my_index/_doc/1
{ "name": "<NAME>" }

PUT my_index/_doc/2
{ "name": "<NAME>" }

GET my_index/_search
{
  "query": {
    "term": {
      "name.length": 3 <3>
    }
  }
}
--------------------------------------------------
// CONSOLE
<1> The `name` field is an analyzed string field which uses the default `standard` analyzer.
<2> The `name.length` field is a `token_count` <<multi-fields,multi-field>> which will index the number of tokens in the `name` field.
<3> This query matches only the document containing `<NAME>`, as it contains three tokens.


[[token-count-params]]
==== Parameters for `token_count` fields

The following parameters are accepted by `token_count` fields:

[horizontal]

<<analyzer,`analyzer`>>::

    The <<analysis,analyzer>> which should be used to analyze the string
    value. Required. For best performance, use an analyzer without token
    filters.

`enable_position_increments`:: 

Indicates if position increments should be counted. 
Set to `false` if you don't want to count tokens removed by analyzer filters (like <<analysis-stop-tokenfilter,`stop`>>). 
Defaults to `true`.

<<mapping-boost,`boost`>>::

    Mapping field-level query time boosting. Accepts a floating point number, defaults
    to `1.0`.

<<doc-values,`doc_values`>>::

    Should the field be stored on disk in a column-stride fashion, so that it
    can later be used for sorting, aggregations, or scripting? Accepts `true`
    (default) or `false`.

<<mapping-index,`index`>>::

    Should the field be searchable? Accepts `true` (default) and `false`.

<<null-value,`null_value`>>::

    Accepts a numeric value of the same `type` as the field which is
{'fraction_non_alphanumeric': 0.16757701551234433, 'fraction_numerical': 0.004806641905178064, 'mean_word_length': 2.694915254237288, 'pattern_counts': {'":': 28, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3011580', 'n_tokens_mistral': 1378, 'n_tokens_neox': 1337, 'n_words': 488}