multiword expression tokenizer c 20012023 nltk project rob malouf rmaloufmail sdsu edu url https www nltk org for license information see license txt multiword expression tokenizer a mwetokenizer takes a string which has already been divided into tokens and retokenizes it merging multiword expressions into single tokens using a lexicon of mwes from nltk tokenize import mwetokenizer tokenizer mwetokenizer a little a little bit a lot tokenizer addmwe in spite of tokenizer tokenize testing testing testing one two three split testing testing testing one two three tokenizer tokenize this is a test in spite split this is a test in spite tokenizer tokenize in a little or a little bit or a lot in spite of split in alittle or alittlebit or alot inspiteof a tokenizer that processes tokenized text and merges multiword expressions into single tokens initialize the multiword tokenizer with a list of expressions and a separator type mwes listliststr param mwes a sequence of multiword expressions to be merged where each mwe is a sequence of strings type separator str param separator string that should be inserted between words in a multiword expression token default is add a multiword expression to the lexicon stored as a word trie we use util trie to represent the trie its form is a dict of dicts the key true marks the end of a valid mwe param mwe the multiword expression we re adding into the word trie type mwe tuplestr or liststr example tokenizer mwetokenizer tokenizer addmwe a b tokenizer addmwe a b c tokenizer addmwe a x expected a x true none b true none c true none tokenizer mwes expected true param text a list containing tokenized text type text liststr return a list of the tokenized text with multiwords merged together rtype liststr example tokenizer mwetokenizer hors d oeuvre separator tokenizer tokenizean hors d oeuvre tonight sir split an horsd oeuvre tonight sir possible mwe match success no match so backtrack multi word expression tokenizer c 2001 2023 nltk project rob malouf rmalouf mail sdsu edu url https www nltk org for license information see license txt multi word expression tokenizer a mwetokenizer takes a string which has already been divided into tokens and retokenizes it merging multi word expressions into single tokens using a lexicon of mwes from nltk tokenize import mwetokenizer tokenizer mwetokenizer a little a little bit a lot tokenizer add_mwe in spite of tokenizer tokenize testing testing testing one two three split testing testing testing one two three tokenizer tokenize this is a test in spite split this is a test in spite tokenizer tokenize in a little or a little bit or a lot in spite of split in a_little or a_little_bit or a_lot in_spite_of a tokenizer that processes tokenized text and merges multi word expressions into single tokens initialize the multi word tokenizer with a list of expressions and a separator type mwes list list str param mwes a sequence of multi word expressions to be merged where each mwe is a sequence of strings type separator str param separator string that should be inserted between words in a multi word expression token default is _ add a multi word expression to the lexicon stored as a word trie we use util trie to represent the trie its form is a dict of dicts the key true marks the end of a valid mwe param mwe the multi word expression we re adding into the word trie type mwe tuple str or list str example tokenizer mwetokenizer tokenizer add_mwe a b tokenizer add_mwe a b c tokenizer add_mwe a x expected a x true none b true none c true none tokenizer _mwes expected true param text a list containing tokenized text type text list str return a list of the tokenized text with multi words merged together rtype list str example tokenizer mwetokenizer hors d oeuvre separator tokenizer tokenize an hors d oeuvre tonight sir split an hors d oeuvre tonight sir possible mwe match and len trie text j 0 success no match so backtrack
from nltk.tokenize.api import TokenizerI
from nltk.util import Trie


class MWETokenizer(TokenizerI):
    

    def __init__(self, mwes=None, separator="_"):
        
        if not mwes:
            mwes = []
        self._mwes = Trie(mwes)
        self._separator = separator

    def add_mwe(self, mwe):
        
        self._mwes.insert(mwe)

    def tokenize(self, text):
        
        i = 0
        n = len(text)
        result = []

        while i < n:
            if text[i] in self._mwes:
                
                j = i
                trie = self._mwes
                last_match = -1
                while j < n and text[j] in trie:  
                    trie = trie[text[j]]
                    j = j + 1
                    if Trie.LEAF in trie:
                        last_match = j
                else:
                    if last_match > -1:
                        j = last_match

                    if Trie.LEAF in trie or last_match > -1:
                        
                        result.append(self._separator.join(text[i:j]))
                        i = j
                    else:
                        
                        result.append(text[i])
                        i += 1
            else:
                result.append(text[i])
                i += 1
        return result