natural language toolkit stemmers c 20012023 nltk project steven tomcavage stomcavalaw upenn edu url https www nltk org for license information see license txt a word stemmer based on the lancaster paicehusk stemming algorithm paice chris d another stemmer acm sigir forum 24 3 1990 5661 lancaster stemmer from nltk stem lancaster import lancasterstemmer st lancasterstemmer st stem maximum remove um when word is intact maxim st stem presumably don t remove um when word is not intact presum st stem multiply no action taken if word ends with ply multiply st stem provision replace sion with j to trigger j set of rules provid st stem owed word starting with vowel must contain at least 2 letters ow st stem ear ditto ear st stem saying words starting with consonant must contain at least 3 say st stem crying letters and one of those letters must be a vowel cry st stem string ditto string st stem meant ditto meant st stem cement ditto cem stpre lancasterstemmerstripprefixflagtrue stpre stem kilometer test prefix met stcustom lancasterstemmerruletuplessen4 s1t stcustom stemness change s to t nest the rule list is static since it doesn t change between instances create an instance of the lancaster stemmer setup an empty rule dictionary this will be filled in later self ruledictionary check if a user wants to strip prefix self stripprefix stripprefixflag check if a user wants to use hisher own rule tuples self ruletuple ruletuple if ruletuple else self defaultruletuple def parserulesself ruletuplenone if there is no argument for the function use class own rule tuple ruletuple ruletuple if ruletuple else self ruletuple validrule re compileraz daz empty any old rules from the rule set before adding new ones self ruledictionary for rule in ruletuple if not validrule matchrule raise valueerrorfthe rule rule is invalid firstletter rule0 1 if firstletter in self ruledictionary self ruledictionaryfirstletter appendrule else self ruledictionaryfirstletter rule def stemself word lowercase the word since all the rules are lowercased save a copy of the original word if rule dictionary is empty parse rule tuple perform the actual word stemming validrule re compileraz daz proceed true while proceed find the position of the last letter of the word to be stemmed lastletterposition self getlastletterword only stem the word if it has a last letter and a rule matching that last letter if lastletterposition 0 or wordlastletterposition not in self ruledictionary proceed false else rulewasapplied false go through each rule that matches the word s final letter for rule in self ruledictionarywordlastletterposition rulematch validrule matchrule if rulematch endingstring intactflag removetotal appendstring contflag rulematch groups convert the number of chars to remove when stemming from a string to an integer removetotal intremovetotal proceed if word s ending matches rule s word ending if word endswithendingstring 1 if intactflag if word intactword and self isacceptable word removetotal word self applyrule word removetotal appendstring rulewasapplied true if contflag proceed false break elif self isacceptableword removetotal word self applyrule word removetotal appendstring rulewasapplied true if contflag proceed false break if no rules apply the word doesn t need any more stemming if rulewasapplied false proceed false return word def getlastletterself word determine if the word is acceptable for stemming wordisacceptable false if the word starts with a vowel it must be at least 2 characters long to be stemmed if word0 in aeiouy if lenword removetotal 2 wordisacceptable true if the word starts with a consonant it must be at least 3 characters long including one vowel to be stemmed elif lenword removetotal 3 if word1 in aeiouy wordisacceptable true elif word2 in aeiouy wordisacceptable true return wordisacceptable def applyruleself word removetotal appendstring remove letters from the end of the word and add new letters to the end of the truncated word remove prefix from a word this function originally taken from whoosh natural language toolkit stemmers c 2001 2023 nltk project steven tomcavage stomcava law upenn edu url https www nltk org for license information see license txt a word stemmer based on the lancaster paice husk stemming algorithm paice chris d another stemmer acm sigir forum 24 3 1990 56 61 lancaster stemmer from nltk stem lancaster import lancasterstemmer st lancasterstemmer st stem maximum remove um when word is intact maxim st stem presumably don t remove um when word is not intact presum st stem multiply no action taken if word ends with ply multiply st stem provision replace sion with j to trigger j set of rules provid st stem owed word starting with vowel must contain at least 2 letters ow st stem ear ditto ear st stem saying words starting with consonant must contain at least 3 say st stem crying letters and one of those letters must be a vowel cry st stem string ditto string st stem meant ditto meant st stem cement ditto cem st_pre lancasterstemmer strip_prefix_flag true st_pre stem kilometer test prefix met st_custom lancasterstemmer rule_tuple ssen4 s1t st_custom stem ness change s to t nest the rule list is static since it doesn t change between instances ia if intact a if intact bb b ytic ys ic nc nt dd d ied y ceed cess eed ee ed hood e lief liev if ing iag y ag gg g th if intact guish ct ish i if intact i y ij id see nois4j vis3j fuj fus uj ud oj od hej her verj vert misj mit nj nd j s ifiabl iabl y abl ibl bil bl cl c iful y ful ul ial ual al ll l ium um if intact ism mm m sion j xion ct ion ian an protect een en nn n ship pp p er protect ear ar or ur rr r tr t ier y ies y sis s is ness protect ss ous us if intact s if intact s s plicat ply at ment ent ant ript rib orpt orb duct duc sumpt sum cept ceiv olut olv protect sist ist tt t iqu ogu og siv j protect eiv iv bly bl ily y protect ply ly ogy og phy ph omy om opy op ity ety lty l istry ary ory ify ncy nt acy iz yz ys create an instance of the lancaster stemmer setup an empty rule dictionary this will be filled in later check if a user wants to strip prefix check if a user wants to use his her own rule tuples validate the set of rules used in this stemmer if this function is called as an individual method without using stem method rule_tuple argument will be compiled into self rule_dictionary if this function is called within stem self _rule_tuple will be used if there is no argument for the function use class own rule tuple empty any old rules from the rule set before adding new ones stem a word using the lancaster stemmer lower case the word since all the rules are lower cased save a copy of the original word if rule dictionary is empty parse rule tuple perform the actual word stemming find the position of the last letter of the word to be stemmed only stem the word if it has a last letter and a rule matching that last letter go through each rule that matches the word s final letter convert the number of chars to remove when stemming from a string to an integer proceed if word s ending matches rule s word ending if no rules apply the word doesn t need any more stemming get the zero based index of the last alphabetic character in this string determine if the word is acceptable for stemming if the word starts with a vowel it must be at least 2 characters long to be stemmed if the word starts with a consonant it must be at least 3 characters long including one vowel to be stemmed apply the stemming rule to the word remove letters from the end of the word and add new letters to the end of the truncated word remove prefix from a word this function originally taken from whoosh
import re

from nltk.stem.api import StemmerI


class LancasterStemmer(StemmerI):
    

    
    default_rule_tuple = (
        "ai*2.",  
        "a*1.",  
        "bb1.",  
        "city3s.",  
        "ci2>",  
        "cn1t>",  
        "dd1.",  
        "dei3y>",  
        "deec2ss.",  
        "dee1.",  
        "de2>",  
        "dooh4>",  
        "e1>",  
        "feil1v.",  
        "fi2>",  
        "gni3>",  
        "gai3y.",  
        "ga2>",  
        "gg1.",  
        "ht*2.",  
        "hsiug5ct.",  
        "hsi3>",  
        "i*1.",  
        "i1y>",  
        "ji1d.",  
        "juf1s.",  
        "ju1d.",  
        "jo1d.",  
        "jeh1r.",  
        "jrev1t.",  
        "jsim2t.",  
        "jn1d.",  
        "j1s.",  
        "lbaifi6.",  
        "lbai4y.",  
        "lba3>",  
        "lbi3.",  
        "lib2l>",  
        "lc1.",  
        "lufi4y.",  
        "luf3>",  
        "lu2.",  
        "lai3>",  
        "lau3>",  
        "la2>",  
        "ll1.",  
        "mui3.",  
        "mu*2.",  
        "msi3>",  
        "mm1.",  
        "nois4j>",  
        "noix4ct.",  
        "noi3>",  
        "nai3>",  
        "na2>",  
        "nee0.",  
        "ne2>",  
        "nn1.",  
        "pihs4>",  
        "pp1.",  
        "re2>",  
        "rae0.",  
        "ra2.",  
        "ro2>",  
        "ru2>",  
        "rr1.",  
        "rt1>",  
        "rei3y>",  
        "sei3y>",  
        "sis2.",  
        "si2>",  
        "ssen4>",  
        "ss0.",  
        "suo3>",  
        "su*2.",  
        "s*1>",  
        "s0.",  
        "tacilp4y.",  
        "ta2>",  
        "tnem4>",  
        "tne3>",  
        "tna3>",  
        "tpir2b.",  
        "tpro2b.",  
        "tcud1.",  
        "tpmus2.",  
        "tpec2iv.",  
        "tulo2v.",  
        "tsis0.",  
        "tsi3>",  
        "tt1.",  
        "uqi3.",  
        "ugo1.",  
        "vis3j>",  
        "vie0.",  
        "vi2>",  
        "ylb1>",  
        "yli3y>",  
        "ylp0.",  
        "yl2>",  
        "ygo1.",  
        "yhp1.",  
        "ymo1.",  
        "ypo1.",  
        "yti3>",  
        "yte3>",  
        "ytl2.",  
        "yrtsi5.",  
        "yra3>",  
        "yro3>",  
        "yfi3.",  
        "ycn2t>",  
        "yca3>",  
        "zi2>",  
        "zy1s.",  
    )

    def __init__(self, rule_tuple=None, strip_prefix_flag=False):
        
        
        self.rule_dictionary = {}
        
        self._strip_prefix = strip_prefix_flag
        
        self._rule_tuple = rule_tuple if rule_tuple else self.default_rule_tuple

    def parseRules(self, rule_tuple=None):
        
        
        rule_tuple = rule_tuple if rule_tuple else self._rule_tuple
        valid_rule = re.compile(r"^[a-z]+\*?\d[a-z]*[>\.]?$")
        
        self.rule_dictionary = {}

        for rule in rule_tuple:
            if not valid_rule.match(rule):
                raise ValueError(f"The rule {rule} is invalid")
            first_letter = rule[0:1]
            if first_letter in self.rule_dictionary:
                self.rule_dictionary[first_letter].append(rule)
            else:
                self.rule_dictionary[first_letter] = [rule]

    def stem(self, word):
        
        
        word = word.lower()
        word = self.__stripPrefix(word) if self._strip_prefix else word

        
        intact_word = word

        
        if not self.rule_dictionary:
            self.parseRules()

        return self.__doStemming(word, intact_word)

    def __doStemming(self, word, intact_word):
        

        valid_rule = re.compile(r"^([a-z]+)(\*?)(\d)([a-z]*)([>\.]?)$")

        proceed = True

        while proceed:

            
            last_letter_position = self.__getLastLetter(word)

            
            if (
                last_letter_position < 0
                or word[last_letter_position] not in self.rule_dictionary
            ):
                proceed = False

            else:
                rule_was_applied = False

                
                for rule in self.rule_dictionary[word[last_letter_position]]:
                    rule_match = valid_rule.match(rule)
                    if rule_match:
                        (
                            ending_string,
                            intact_flag,
                            remove_total,
                            append_string,
                            cont_flag,
                        ) = rule_match.groups()

                        
                        
                        remove_total = int(remove_total)

                        
                        if word.endswith(ending_string[::-1]):
                            if intact_flag:
                                if word == intact_word and self.__isAcceptable(
                                    word, remove_total
                                ):
                                    word = self.__applyRule(
                                        word, remove_total, append_string
                                    )
                                    rule_was_applied = True
                                    if cont_flag == ".":
                                        proceed = False
                                    break
                            elif self.__isAcceptable(word, remove_total):
                                word = self.__applyRule(
                                    word, remove_total, append_string
                                )
                                rule_was_applied = True
                                if cont_flag == ".":
                                    proceed = False
                                break
                
                if rule_was_applied == False:
                    proceed = False
        return word

    def __getLastLetter(self, word):
        
        last_letter = -1
        for position in range(len(word)):
            if word[position].isalpha():
                last_letter = position
            else:
                break
        return last_letter

    def __isAcceptable(self, word, remove_total):
        
        word_is_acceptable = False
        
        
        if word[0] in "aeiouy":
            if len(word) - remove_total >= 2:
                word_is_acceptable = True
        
        
        elif len(word) - remove_total >= 3:
            if word[1] in "aeiouy":
                word_is_acceptable = True
            elif word[2] in "aeiouy":
                word_is_acceptable = True
        return word_is_acceptable

    def __applyRule(self, word, remove_total, append_string):
        
        
        new_word_length = len(word) - remove_total
        word = word[0:new_word_length]

        
        if append_string:
            word += append_string
        return word

    def __stripPrefix(self, word):
        
        for prefix in (
            "kilo",
            "micro",
            "milli",
            "intra",
            "ultra",
            "mega",
            "nano",
            "pico",
            "pseudo",
        ):
            if word.startswith(prefix):
                return word[len(prefix) :]
        return word

    def __repr__(self):
        return "<LancasterStemmer>"