natural language toolkit python port of the mtevalv14 pl tokenizer c 20012015 nltk project liling tan ported from ftp jaguar ncsl nist govmtresourcesmtevalv14 pl contributors ozan caglayan wiktor stribizew url https www nltk org for license information see license txt this is a nltk port of the tokenizer used in the nist bleu evaluation script https github commosessmtmosesdecoderblobmasterscriptsgenericmtevalv14 pll926 which was also ported into python in https github comliumlstnmtpyblobmasternmtpymetricsmtevalbleu pyl162 this nist tokenizer is sentencebased instead of the original paragraphbased tokenization from mteval14 pl the sentencebased tokenization is consistent with the other tokenizers available in nltk from nltk tokenize nist import nisttokenizer nist nisttokenizer s good muffins cost 3 88 in new york expectedlower u good u muffins u cost u u 3 88 u in u new u york u expectedcased u good u muffins u cost u u 3 88 u in u new u york u nist tokenizes lowercasefalse expectedcased true nist tokenizes lowercasetrue expectedlower lowercased true the internationaltokenize is the preferred function when tokenizing noneuropean text e g from nltk tokenize nist import nisttokenizer nist nisttokenizer input strings albb u alibaba group holding limited chinese us a chinese ecommerce company amz u amazon com inc mzn is an american electronic commerce rkt u rakuten inc rakuten kabushikigaisha is a japanese electronic commerce and internet company based in tokyo expected tokens expectedalbb u alibaba u group u holding u limited u u chinese u u u963fu91ccu5df4u5df4u96c6u56e2u63a7u80a1 u u6709u9650u516cu53f8 u expectedamz u amazon u u com u u inc u u u u u02c8xe6 u m expectedrkt u rakuten u u inc u u u u697du5929u682au5f0fu4f1au793e u rakuten u kabushiki u u gaisha nist internationaltokenizealbb 10 expectedalbb true nist internationaltokenizeamz 10 expectedamz true nist internationaltokenizerkt 10 expectedrkt true doctest for patching issue 1926 sent u this is a foou2604sentence expectedsent u this u is u a u foo u u2604 u sentence u nist internationaltokenizesent expectedsent true strip skipped tags strip endofline hyphenation and join lines tokenize punctuation tokenize period and comma unless preceded by a digit tokenize period and comma unless followed by a digit tokenize dash when preceded by a digit perluniprops characters used in nist tokenizer python regexes needs to escape some special symbols see see https stackoverflow comq45670950610569 note in the original perl implementation pz and pzl were used to i strip trailing and heading spaces and ii dedeuplicate spaces in python this would do joinstr strip split thus the next two lines were commented out lineseparator str joinperluniprops chars lineseparator i e pzl separator str joinperluniprops chars separator i e pz pads nonascii strings with space tokenize any punctuation unless followed and preceded by a digit tokenize symbols performs the language independent string substituitions it s a strange order of regexes it ll be better to unescape after stripeolhyphen but let s keep it close to the original nist implementation regexp substitution self stripskip text regexp subsubstitution text text xmlunescapetext regexp substitution self stripeolhyphen text regexp subsubstitution text return text def tokenizeself text lowercasefalse westernlangtrue returnstrfalse text strtext language independent regex text self langindependentsubtext language dependent regex if westernlang pad string with whitespace text text if lowercase text text lower for regexp substitution in self langdependentregexes text regexp subsubstitution text remove contiguous whitespaces text jointext split finally strips heading and trailing spaces and converts output string into unicode text strtext strip return text if returnstr else text split def internationaltokenize self text lowercasefalse splitnonasciitrue returnstrfalse text strtext different from the normal tokenize stripeolhyphen is applied first before unescaping regexp substitution self stripskip text regexp subsubstitution text regexp substitution self stripeolhyphen text regexp subsubstitution text text xmlunescapetext if lowercase text text lower for regexp substitution in self internationalregexes text regexp subsubstitution text make sure that there s only one space only between words strip leading and trailing spaces text jointext strip split return text if returnstr else text split natural language toolkit python port of the mteval v14 pl tokenizer c 2001 2015 nltk project liling tan ported from ftp jaguar ncsl nist gov mt resources mteval v14 pl contributors ozan caglayan wiktor stribizew url https www nltk org for license information see license txt this is a nltk port of the tokenizer used in the nist bleu evaluation script https github com moses smt mosesdecoder blob master scripts generic mteval v14 pl l926 which was also ported into python in https github com lium lst nmtpy blob master nmtpy metrics mtevalbleu py l162 this nist tokenizer is sentence based instead of the original paragraph based tokenization from mteval 14 pl the sentence based tokenization is consistent with the other tokenizers available in nltk from nltk tokenize nist import nisttokenizer nist nisttokenizer s good muffins cost 3 88 in new york expected_lower u good u muffins u cost u u 3 88 u in u new u york u expected_cased u good u muffins u cost u u 3 88 u in u new u york u nist tokenize s lowercase false expected_cased true nist tokenize s lowercase true expected_lower lowercased true the international_tokenize is the preferred function when tokenizing non european text e g from nltk tokenize nist import nisttokenizer nist nisttokenizer input strings albb u alibaba group holding limited chinese 阿里巴巴集团控股 有限公司 us a chinese e commerce company amz u amazon com inc ˈæməzɒn is an american electronic commerce rkt u rakuten inc 楽天株式会社 rakuten kabushiki gaisha is a japanese electronic commerce and internet company based in tokyo expected tokens expected_albb u alibaba u group u holding u limited u u chinese u u u963f u91cc u5df4 u5df4 u96c6 u56e2 u63a7 u80a1 u u6709 u9650 u516c u53f8 u expected_amz u amazon u u com u u inc u u u u u02c8 xe6 u m expected_rkt u rakuten u u inc u u u u697d u5929 u682a u5f0f u4f1a u793e u rakuten u kabushiki u u gaisha nist international_tokenize albb 10 expected_albb true nist international_tokenize amz 10 expected_amz true nist international_tokenize rkt 10 expected_rkt true doctest for patching issue 1926 sent u this is a foo u2604sentence expected_sent u this u is u a u foo u u2604 u sentence u nist international_tokenize sent expected_sent true strip skipped tags strip end of line hyphenation and join lines tokenize punctuation tokenize period and comma unless preceded by a digit tokenize period and comma unless followed by a digit tokenize dash when preceded by a digit perluniprops characters used in nist tokenizer i e p n i e p p i e p s python regexes needs to escape some special symbols see see https stackoverflow com q 45670950 610569 note in the original perl implementation p z and p zl were used to i strip trailing and heading spaces and ii de deuplicate spaces in python this would do join str strip split thus the next two lines were commented out line_separator str join perluniprops chars line_separator i e p zl separator str join perluniprops chars separator i e p z pads non ascii strings with space tokenize any punctuation unless followed and preceded by a digit tokenize symbols performs the language independent string substituitions it s a strange order of regexes it ll be better to unescape after strip_eol_hyphen but let s keep it close to the original nist implementation language independent regex language dependent regex pad string with whitespace remove contiguous whitespaces finally strips heading and trailing spaces and converts output string into unicode different from the normal tokenize strip_eol_hyphen is applied first before unescaping make sure that there s only one space only between words strip leading and trailing spaces
import io
import re

from nltk.corpus import perluniprops
from nltk.tokenize.api import TokenizerI
from nltk.tokenize.util import xml_unescape


class NISTTokenizer(TokenizerI):
    

    
    STRIP_SKIP = re.compile("<skipped>"), ""
    
    STRIP_EOL_HYPHEN = re.compile("\u2028"), " "
    
    PUNCT = re.compile(r"([\{-\~\[-\` -\&\(-\+\:-\@\/])"), " \\1 "
    
    PERIOD_COMMA_PRECEED = re.compile(r"([^0-9])([\.,])"), "\\1 \\2 "
    
    PERIOD_COMMA_FOLLOW = re.compile(r"([\.,])([^0-9])"), " \\1 \\2"
    
    DASH_PRECEED_DIGIT = re.compile("([0-9])(-)"), "\\1 \\2 "

    LANG_DEPENDENT_REGEXES = [
        PUNCT,
        PERIOD_COMMA_PRECEED,
        PERIOD_COMMA_FOLLOW,
        DASH_PRECEED_DIGIT,
    ]

    
    pup_number = str("".join(set(perluniprops.chars("Number"))))  
    pup_punct = str("".join(set(perluniprops.chars("Punctuation"))))  
    pup_symbol = str("".join(set(perluniprops.chars("Symbol"))))  

    
    
    number_regex = re.sub(r"[]^\\-]", r"\\\g<0>", pup_number)
    punct_regex = re.sub(r"[]^\\-]", r"\\\g<0>", pup_punct)
    symbol_regex = re.sub(r"[]^\\-]", r"\\\g<0>", pup_symbol)

    
    
    
    
    
    
    

    
    NONASCII = re.compile("([\x00-\x7f]+)"), r" \1 "
    
    PUNCT_1 = (
        re.compile(f"([{number_regex}])([{punct_regex}])"),
        "\\1 \\2 ",
    )
    PUNCT_2 = (
        re.compile(f"([{punct_regex}])([{number_regex}])"),
        " \\1 \\2",
    )
    
    SYMBOLS = re.compile(f"([{symbol_regex}])"), " \\1 "

    INTERNATIONAL_REGEXES = [NONASCII, PUNCT_1, PUNCT_2, SYMBOLS]

    def lang_independent_sub(self, text):
        
        
        
        
        regexp, substitution = self.STRIP_SKIP
        text = regexp.sub(substitution, text)
        text = xml_unescape(text)
        regexp, substitution = self.STRIP_EOL_HYPHEN
        text = regexp.sub(substitution, text)
        return text

    def tokenize(self, text, lowercase=False, western_lang=True, return_str=False):
        text = str(text)
        
        text = self.lang_independent_sub(text)
        
        if western_lang:
            
            text = " " + text + " "
            if lowercase:
                text = text.lower()
            for regexp, substitution in self.LANG_DEPENDENT_REGEXES:
                text = regexp.sub(substitution, text)
        
        text = " ".join(text.split())
        
        
        text = str(text.strip())
        return text if return_str else text.split()

    def international_tokenize(
        self, text, lowercase=False, split_non_ascii=True, return_str=False
    ):
        text = str(text)
        
        
        regexp, substitution = self.STRIP_SKIP
        text = regexp.sub(substitution, text)
        regexp, substitution = self.STRIP_EOL_HYPHEN
        text = regexp.sub(substitution, text)
        text = xml_unescape(text)

        if lowercase:
            text = text.lower()

        for regexp, substitution in self.INTERNATIONAL_REGEXES:
            text = regexp.sub(substitution, text)

        
        
        text = " ".join(text.strip().split())
        return text if return_str else text.split()