<commit_before>---
title: Tokenizer
teaser: Segment text into words, punctuations marks etc.
tag: class
source: spacy/tokenizer.pyx
---

Segment text, and create `Doc` objects with the discovered segment boundaries.

## Tokenizer.\_\_init\_\_ {#init tag="method"}

Create a `Tokenizer`, to create `Doc` objects given unicode text.

> #### Example
>
> ```python
> # Construction 1
> from spacy.tokenizer import Tokenizer
> from spacy.lang.en import English
> nlp = English()
> tokenizer = Tokenizer(nlp.vocab)
>
> # Construction 2
> from spacy.lang.en import English
> nlp = English()
> tokenizer = nlp.Defaults.create_tokenizer(nlp)
> ```

| Name             | Type        | Description                                                                         |
| ---------------- | ----------- | ----------------------------------------------------------------------------------- |
| `vocab`          | `Vocab`     | A storage container for lexical types.                                              |
| `rules`          | dict        | Exceptions and special-cases for the tokenizer.                                     |
| `prefix_search`  | callable    | A function matching the signature of `re.compile(string).search` to match prefixes. |
| `suffix_search`  | callable    | A function matching the signature of `re.compile(string).search` to match suffixes. |
| `infix_finditer` | callable    | A function matching the signature of `re.compile(string).finditer` to find infixes. |
| `token_match`    | callable    | A boolean function matching strings to be recognized as tokens.                     |
| **RETURNS**      | `Tokenizer` | The newly constructed object.                                                       |

## Tokenizer.\_\_call\_\_ {#call tag="method"}

Tokenize a string.

> #### Example
>
> ```python
> tokens = tokenizer(u"This is a sentence")
> assert len(tokens) == 4
> ```

| Name        | Type    | Description                             |
| ----------- | ------- | --------------------------------------- |
| `string`    | unicode | The string to tokenize.                 |
| **RETURNS** | `Doc`   | A container for linguistic annotations. |

## Tokenizer.pipe {#pipe tag="method"}

Tokenize a stream of texts.

> #### Example
>
> ```python
> texts = [u"One document.", u"...", u"Lots of documents"]
> for doc in tokenizer.pipe(texts, batch_size=50):
>     pass
> ```

| Name         | Type  | Description                                              |
| ------------ | ----- | -------------------------------------------------------- |
| `texts`      | -     | A sequence of unicode texts.                             |
| `batch_size` | int   | The number of texts to accumulate in an internal buffer. Defaults to `1000`.|
| **YIELDS**   | `Doc` | A sequence of Doc objects, in order.                     |

## Tokenizer.find_infix {#find_infix tag="method"}

Find internal split points of the string.

| Name        | Type    | Description                                                                                                                                        |
| ----------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `string`    | unicode | The string to split.                                                                                                                               |
| **RETURNS** | list    | A list of `re.MatchObject` objects that have `.start()` and `.end()` methods, denoting the placement of internal segment separators, e.g. hyphens. |

## Tokenizer.find_prefix {#find_prefix tag="method"}

Find the length of a prefix that should be segmented from the string, or `None`
if no prefix rules match.

| Name        | Type    | Description                                            |
| ----------- | ------- | ------------------------------------------------------ |
| `string`    | unicode | The string to segment.                                 |
| **RETURNS** | int     | The length of the prefix if present, otherwise `None`. |

## Tokenizer.find_suffix {#find_suffix tag="method"}

Find the length of a suffix that should be segmented from the string, or `None`
if no suffix rules match.

| Name        | Type         | Description                                            |<commit_msg>Auto-format [ci skip]<commit_after>---
title: Tokenizer
teaser: Segment text into words, punctuations marks etc.
tag: class
source: spacy/tokenizer.pyx
---

Segment text, and create `Doc` objects with the discovered segment boundaries.

## Tokenizer.\_\_init\_\_ {#init tag="method"}

Create a `Tokenizer`, to create `Doc` objects given unicode text.

> #### Example
>
> ```python
> # Construction 1
> from spacy.tokenizer import Tokenizer
> from spacy.lang.en import English
> nlp = English()
> tokenizer = Tokenizer(nlp.vocab)
>
> # Construction 2
> from spacy.lang.en import English
> nlp = English()
> tokenizer = nlp.Defaults.create_tokenizer(nlp)
> ```

| Name             | Type        | Description                                                                         |
| ---------------- | ----------- | ----------------------------------------------------------------------------------- |
| `vocab`          | `Vocab`     | A storage container for lexical types.                                              |
| `rules`          | dict        | Exceptions and special-cases for the tokenizer.                                     |
| `prefix_search`  | callable    | A function matching the signature of `re.compile(string).search` to match prefixes. |
| `suffix_search`  | callable    | A function matching the signature of `re.compile(string).search` to match suffixes. |
| `infix_finditer` | callable    | A function matching the signature of `re.compile(string).finditer` to find infixes. |
| `token_match`    | callable    | A boolean function matching strings to be recognized as tokens.                     |
| **RETURNS**      | `Tokenizer` | The newly constructed object.                                                       |

## Tokenizer.\_\_call\_\_ {#call tag="method"}

Tokenize a string.

> #### Example
>
> ```python
> tokens = tokenizer(u"This is a sentence")
> assert len(tokens) == 4
> ```

| Name        | Type    | Description                             |
| ----------- | ------- | --------------------------------------- |
| `string`    | unicode | The string to tokenize.                 |
| **RETURNS** | `Doc`   | A container for linguistic annotations. |

## Tokenizer.pipe {#pipe tag="method"}

Tokenize a stream of texts.

> #### Example
>
> ```python
> texts = [u"One document.", u"...", u"Lots of documents"]
> for doc in tokenizer.pipe(texts, batch_size=50):
>     pass
> ```

| Name         | Type  | Description                                                                  |
| ------------ | ----- | ---------------------------------------------------------------------------- |
| `texts`      | -     | A sequence of unicode texts.                                                 |
| `batch_size` | int   | The number of texts to accumulate in an internal buffer. Defaults to `1000`. |
| **YIELDS**   | `Doc` | A sequence of Doc objects, in order.                                         |

## Tokenizer.find_infix {#find_infix tag="method"}

Find internal split points of the string.

| Name        | Type    | Description                                                                                                                                        |
| ----------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `string`    | unicode | The string to split.                                                                                                                               |
| **RETURNS** | list    | A list of `re.MatchObject` objects that have `.start()` and `.end()` methods, denoting the placement of internal segment separators, e.g. hyphens. |

## Tokenizer.find_prefix {#find_prefix tag="method"}

Find the length of a prefix that should be segmented from the string, or `None`
if no prefix rules match.

| Name        | Type    | Description                                            |
| ----------- | ------- | ------------------------------------------------------ |
| `string`    | unicode | The string to segment.                                 |
| **RETURNS** | int     | The length of the prefix if present, otherwise `None`. |

## Tokenizer.find_suffix {#find_suffix tag="method"}

Find the length of a suffix that should be segmented from the string, or `None`
if no suffix rules match.

| Name        | Type         | Description                                            |
{'fraction_non_alphanumeric': 0.20608450704225353, 'fraction_numerical': 0.0020281690140845072, 'mean_word_length': 1.1943139678615575, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 54, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '751428', 'n_tokens_mistral': 2054, 'n_tokens_neox': 2002, 'n_words': 674}