natural language toolkit tokenizer interface c 20012023 nltk project edward loper edlopergmail com steven bird stevenbird1gmail com url https www nltk org for license information see license txt tokenizer interface a processing interface for tokenizing a string subclasses must define tokenize or tokenizesents or both return a tokenized copy of s rtype liststr identify the tokens using integer offsets starti endi where sstarti endi is the corresponding token rtype iteratortupleint int apply self tokenize to each element of strings i e return self tokenizes for s in strings rtype listliststr apply self spantokenize to each element of strings i e return self spantokenizes for s in strings yield listtupleint int a tokenizer that divides a string into substrings by splitting on the specified string defined in subclasses natural language toolkit tokenizer interface c 2001 2023 nltk project edward loper edloper gmail com steven bird stevenbird1 gmail com url https www nltk org for license information see license txt tokenizer interface a processing interface for tokenizing a string subclasses must define tokenize or tokenize_sents or both return a tokenized copy of s rtype list str identify the tokens using integer offsets start_i end_i where s start_i end_i is the corresponding token rtype iterator tuple int int apply self tokenize to each element of strings i e return self tokenize s for s in strings rtype list list str apply self span_tokenize to each element of strings i e return self span_tokenize s for s in strings yield list tuple int int a tokenizer that divides a string into substrings by splitting on the specified string defined in subclasses
from abc import ABC, abstractmethod
from typing import Iterator, List, Tuple

from nltk.internals import overridden
from nltk.tokenize.util import string_span_tokenize


class TokenizerI(ABC):
    

    @abstractmethod
    def tokenize(self, s: str) -> List[str]:
        
        if overridden(self.tokenize_sents):
            return self.tokenize_sents([s])[0]

    def span_tokenize(self, s: str) -> Iterator[Tuple[int, int]]:
        
        raise NotImplementedError()

    def tokenize_sents(self, strings: List[str]) -> List[List[str]]:
        
        return [self.tokenize(s) for s in strings]

    def span_tokenize_sents(
        self, strings: List[str]
    ) -> Iterator[List[Tuple[int, int]]]:
        
        for s in strings:
            yield list(self.span_tokenize(s))


class StringTokenizer(TokenizerI):
    

    @property
    @abstractmethod
    def _string(self):
        raise NotImplementedError

    def tokenize(self, s):
        return s.split(self._string)

    def span_tokenize(self, s):
        yield from string_span_tokenize(s, self._string)