<commit_before>//- 💫 DOCS > USAGE > TOKENIZER

include ../../_includes/_mixins

p
    |  Tokenization is the task of splitting a text into meaningful segments,
    |  called #[em tokens].  The input to the tokenizer is a unicode text, and
    |  the output is a #[+api("doc") #[code Doc]] object. To construct a
    |  #[code Doc] object, you need a #[+api("vocab") #[code Vocab]] instance,
    |  a sequence of #[code word] strings, and optionally a sequence of
    |  #[code spaces] booleans, which allow you to maintain alignment of the
    |  tokens into the original string.

+aside("See Also")
    |  If you haven't read up on spaCy's #[+a("data-model") data model] yet,
    |  you should probably have a look. The main point to keep in mind is that
    |  spaCy's #[code Doc] doesn't copy or refer to the original string. The
    |  string is reconstructed from the tokens when required.


+h(2, "special-cases") Adding special case tokenization rules

p
    |  Most domains have at least some idiosyncracies that require custom
    |  tokenization rules. Here's how to add a special case rule to an existing
    |  #[+api("tokenizer") #[code Tokenizer]] instance:

+code.
    nlp = spacy.load('en')
    assert [w.text for w in nlp(u'gimme that')] == [u'gimme', u'that']
    nlp.tokenizer.add_special_case(u'gimme',
        [
            {
                ORTH: u'gim',
                LEMMA: u'give',
                POS: u'VERB'},
            {
                ORTH: u'me'}])
    assert [w.text for w in nlp(u'gimme that')] == [u'gim', u'me', u'that']
    assert [w.lemma_ for w in nlp(u'gimme that')] == [u'give', u'-PRON-', u'that']

p
    |  The special case doesn't have to match an entire whitespace-delimited
    |  substring. The tokenizer will incrementally split off punctuation, and
    |  keep looking up the remaining substring:

+code.
    assert 'gimme' not in [w.text for w in nlp(u'gimme!')]
    assert 'gimme' not in [w.text for w in nlp(u'("...gimme...?")')]

p
    |  The special case rules have precedence over the punctuation splitting:

+code.
    nlp.tokenizer.add_special_case(u"...gimme...?",
        [{
            ORTH: u'...gimme...?", LEMMA: "give", TAG: "VB"}])
    assert len(nlp(u'...gimme...?')) == 1

p
    |  Because the special-case rules allow you to set arbitrary token
    |  attributes, such as the part-of-speech, lemma, etc, they make a good
    |  mechanism for arbitrary fix-up rules. Having this logic live in the<commit_msg>Fix Custom Tokenizer docs<commit_after>//- 💫 DOCS > USAGE > TOKENIZER

include ../../_includes/_mixins

p
    |  Tokenization is the task of splitting a text into meaningful segments,
    |  called #[em tokens].  The input to the tokenizer is a unicode text, and
    |  the output is a #[+api("doc") #[code Doc]] object. To construct a
    |  #[code Doc] object, you need a #[+api("vocab") #[code Vocab]] instance,
    |  a sequence of #[code word] strings, and optionally a sequence of
    |  #[code spaces] booleans, which allow you to maintain alignment of the
    |  tokens into the original string.

+aside("See Also")
    |  If you haven't read up on spaCy's #[+a("data-model") data model] yet,
    |  you should probably have a look. The main point to keep in mind is that
    |  spaCy's #[code Doc] doesn't copy or refer to the original string. The
    |  string is reconstructed from the tokens when required.


+h(2, "special-cases") Adding special case tokenization rules

p
    |  Most domains have at least some idiosyncracies that require custom
    |  tokenization rules. Here's how to add a special case rule to an existing
    |  #[+api("tokenizer") #[code Tokenizer]] instance:

+code.
    import spacy
    from spacy.symbols import ORTH, LEMMA, POS

    nlp = spacy.load('en')
    assert [w.text for w in nlp(u'gimme that')] == [u'gimme', u'that']
    nlp.tokenizer.add_special_case(u'gimme',
        [
            {
                ORTH: u'gim',
                LEMMA: u'give',
                POS: u'VERB'},
            {
                ORTH: u'me'}])
    assert [w.text for w in nlp(u'gimme that')] == [u'gim', u'me', u'that']
    assert [w.lemma_ for w in nlp(u'gimme that')] == [u'give', u'me', u'that']

p
    |  The special case doesn't have to match an entire whitespace-delimited
    |  substring. The tokenizer will incrementally split off punctuation, and
    |  keep looking up the remaining substring:

+code.
    assert 'gimme' not in [w.text for w in nlp(u'gimme!')]
    assert 'gimme' not in [w.text for w in nlp(u'("...gimme...?")')]

p
    |  The special case rules have precedence over the punctuation splitting:

+code.
    nlp.tokenizer.add_special_case(u'...gimme...?',
        [{
            ORTH: u'...gimme...?', LEMMA: u'give', TAG: u'VB'}])
    assert len(nlp(u'...gimme...?')) == 1

p
    |  Because the special-case rules allow you to set arbitrary token
    |  attributes, such as the part-of-speech, lemma, etc, they make a good
    |  mechanism for arbitrary fix-up rules. Having this logic live in the
{'fraction_non_alphanumeric': 0.12656467315716272, 'fraction_numerical': 0.0007947546195112259, 'mean_word_length': 2.7014705882352943, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6193327', 'n_tokens_mistral': 1715, 'n_tokens_neox': 1639, 'n_words': 622}