natural language toolkit tokenizers c 20012023 nltk project edward loper edlopergmail com steven bird stevenbird1gmail com trevor cohn tacohncsse unimelb edu au url https www nltk org for license information see license txt import re from nltk tokenize api import tokenizeri from nltk tokenize util import regexpspantokenize class regexptokenizertokenizeri r a tokenizer that splits a string using a regular expression which matches either the tokens or the separators between tokens tokenizer regexptokenizerr wd s type pattern str param pattern the pattern used to build this tokenizer this pattern must not contain capturing parentheses use noncapturing parentheses e g instead type gaps bool param gaps true if this tokenizer s pattern should be used to find separators between tokens false if this tokenizer s pattern should be used to find the tokens themselves type discardempty bool param discardempty true if any empty tokens generated by the tokenizer should be discarded empty tokens can only be generated if gaps true type flags int param flags the regexp flags used to compile this tokenizer s pattern by default the following flags are used re unicode re multiline re dotall if they gave us a regexp object extract the pattern if our regexp matches gaps use re split if our regexp matches tokens use re findall def initself regexptokenizer initself rs gapstrue class blanklinetokenizerregexptokenizer def initself regexptokenizer initself rsnsns gapstrue class wordpuncttokenizerregexptokenizer r tokenize a text into a sequence of alphabetic and nonalphabetic characters using the regexp wws from nltk tokenize import wordpuncttokenizer s good muffins cost 3 88nin new york please buy mentwo of them nnthanks wordpuncttokenizer tokenizes doctest normalizewhitespace good muffins cost 3 88 in new york please buy me two of them thanks tokenization functions return a tokenized copy of text see class regexptokenizer for descriptions of the arguments natural language toolkit tokenizers c 2001 2023 nltk project edward loper edloper gmail com steven bird stevenbird1 gmail com trevor cohn tacohn csse unimelb edu au url https www nltk org for license information see license txt regular expression tokenizers a regexptokenizer splits a string into substrings using a regular expression for example the following tokenizer forms tokens out of alphabetic sequences money expressions and any other non whitespace sequences from nltk tokenize import regexptokenizer s good muffins cost 3 88 nin new york please buy me ntwo of them n nthanks tokenizer regexptokenizer r w d s tokenizer tokenize s doctest normalize_whitespace good muffins cost 3 88 in new york please buy me two of them thanks a regexptokenizer can use its regexp to match delimiters instead tokenizer regexptokenizer r s gaps true tokenizer tokenize s doctest normalize_whitespace good muffins cost 3 88 in new york please buy me two of them thanks note that empty tokens are not returned when the delimiter appears at the start or end of the string the material between the tokens is discarded for example the following tokenizer selects just the capitalized words capword_tokenizer regexptokenizer r a z w capword_tokenizer tokenize s good new york please thanks this module contains several subclasses of regexptokenizer that use pre defined regular expressions from nltk tokenize import blanklinetokenizer uses s n s n s blanklinetokenizer tokenize s doctest normalize_whitespace good muffins cost 3 88 nin new york please buy me ntwo of them thanks all of the regular expression tokenizers are also available as functions from nltk tokenize import regexp_tokenize wordpunct_tokenize blankline_tokenize regexp_tokenize s pattern r w d s doctest normalize_whitespace good muffins cost 3 88 in new york please buy me two of them thanks wordpunct_tokenize s doctest normalize_whitespace good muffins cost 3 88 in new york please buy me two of them thanks blankline_tokenize s good muffins cost 3 88 nin new york please buy me ntwo of them thanks caution the function regexp_tokenize takes the text as its first argument and the regular expression pattern as its second argument this differs from the conventions used by python s re functions where the pattern is always the first argument this is for consistency with the other nltk tokenizers a tokenizer that splits a string using a regular expression which matches either the tokens or the separators between tokens tokenizer regexptokenizer r w d s type pattern str param pattern the pattern used to build this tokenizer this pattern must not contain capturing parentheses use non capturing parentheses e g instead type gaps bool param gaps true if this tokenizer s pattern should be used to find separators between tokens false if this tokenizer s pattern should be used to find the tokens themselves type discard_empty bool param discard_empty true if any empty tokens generated by the tokenizer should be discarded empty tokens can only be generated if _gaps true type flags int param flags the regexp flags used to compile this tokenizer s pattern by default the following flags are used re unicode re multiline re dotall if they gave us a regexp object extract the pattern if our regexp matches gaps use re split if our regexp matches tokens use re findall tokenize a string on whitespace space tab newline in general users should use the string split method instead from nltk tokenize import whitespacetokenizer s good muffins cost 3 88 nin new york please buy me ntwo of them n nthanks whitespacetokenizer tokenize s doctest normalize_whitespace good muffins cost 3 88 in new york please buy me two of them thanks tokenize a string treating any sequence of blank lines as a delimiter blank lines are defined as lines containing no characters except for space or tab characters tokenize a text into a sequence of alphabetic and non alphabetic characters using the regexp w w s from nltk tokenize import wordpuncttokenizer s good muffins cost 3 88 nin new york please buy me ntwo of them n nthanks wordpuncttokenizer tokenize s doctest normalize_whitespace good muffins cost 3 88 in new york please buy me two of them thanks tokenization functions return a tokenized copy of text see class regexptokenizer for descriptions of the arguments
r

import re

from nltk.tokenize.api import TokenizerI
from nltk.tokenize.util import regexp_span_tokenize


class RegexpTokenizer(TokenizerI):
    r

    def __init__(
        self,
        pattern,
        gaps=False,
        discard_empty=True,
        flags=re.UNICODE | re.MULTILINE | re.DOTALL,
    ):
        
        pattern = getattr(pattern, "pattern", pattern)

        self._pattern = pattern
        self._gaps = gaps
        self._discard_empty = discard_empty
        self._flags = flags
        self._regexp = None

    def _check_regexp(self):
        if self._regexp is None:
            self._regexp = re.compile(self._pattern, self._flags)

    def tokenize(self, text):
        self._check_regexp()
        
        if self._gaps:
            if self._discard_empty:
                return [tok for tok in self._regexp.split(text) if tok]
            else:
                return self._regexp.split(text)

        
        else:
            return self._regexp.findall(text)

    def span_tokenize(self, text):
        self._check_regexp()

        if self._gaps:
            for left, right in regexp_span_tokenize(text, self._regexp):
                if not (self._discard_empty and left == right):
                    yield left, right
        else:
            for m in re.finditer(self._regexp, text):
                yield m.span()

    def __repr__(self):
        return "{}(pattern={!r}, gaps={!r}, discard_empty={!r}, flags={!r})".format(
            self.__class__.__name__,
            self._pattern,
            self._gaps,
            self._discard_empty,
            self._flags,
        )


class WhitespaceTokenizer(RegexpTokenizer):
    r

    def __init__(self):
        RegexpTokenizer.__init__(self, r"\s+", gaps=True)


class BlanklineTokenizer(RegexpTokenizer):
    

    def __init__(self):
        RegexpTokenizer.__init__(self, r"\s*\n\s*\n\s*", gaps=True)


class WordPunctTokenizer(RegexpTokenizer):
    r

    def __init__(self):
        RegexpTokenizer.__init__(self, r"\w+|[^\w\s]+")







def regexp_tokenize(
    text,
    pattern,
    gaps=False,
    discard_empty=True,
    flags=re.UNICODE | re.MULTILINE | re.DOTALL,
):
    
    tokenizer = RegexpTokenizer(pattern, gaps, discard_empty, flags)
    return tokenizer.tokenize(text)


blankline_tokenize = BlanklineTokenizer().tokenize
wordpunct_tokenize = WordPunctTokenizer().tokenize