natural language toolkit the isri arabic stemmer c 20012023 nltk project algorithm kazem taghva rania elkhoury and jeffrey coombs 2005 hosam algasaier hosamhmeyahoo com url https www nltk org for license information see license txt isri arabic stemmer the algorithm for this stemmer is described in taghva k elkoury r and coombs j 2005 arabic stemming without a root dictionary information science research institute university of nevada las vegas usa the information science research institutes isri arabic stemmer shares many features with the khoja stemmer however the main difference is that isri stemmer does not use root dictionary also if a root is not found isri stemmer returned normalized form rather than returning the original unmodified word additional adjustments were made to improve the algorithm 1 adding 60 stop words 2 adding the pattern to isri pattern set 3 the step 2 in the original algorithm was normalizing all hamza this step is discarded because it increases the word ambiguities and changes the original root isri arabic stemmer based on algorithm arabic stemming without a root dictionary information science research institute university of nevada las vegas usa a few minor modifications have been made to isri basic algorithm see the source code of this module for more information isri stemtoken returns arabic root for the given token the isri stemmer requires that all tokens have unicode string types if you use python idle on arabic windows you have to decode text first using arabic 1256 coding length three prefixes length two prefixes length one prefixes length three suffixes length two suffixes length one suffixes groups of length four patterns groups of length five patterns and length three roots stemming a word token using the isri stemmer if 4 word length 7 then stem otherwise no stemming normalization num1 normalize diacritics num2 normalize initial hamza num3 both 12 remove length three and length two prefixes in this order if lenword 6 for pre3 in self p3 if word startswithpre3 return word3 if lenword 5 for pre2 in self p2 if word startswithpre2 return word2 return word def suf32self word remove connective if it precedes a word beginning with if lenword 4 and word 2 u0648u0648 word word1 return word def prow4self word process length five patterns and extract length three roots if word2 in self pr530 and word0 u0627 word word1 word3 elif word3 in self pr531 and word0 u0645 word word1 3 word4 elif word0 in self pr532 and word4 u0629 word word1 4 elif word0 in self pr533 and word2 u062a word word1 word3 elif word0 in self pr534 and word2 u0627 word word1 word3 elif word2 in self pr535 and word4 u0629 word word 2 word3 elif word0 in self pr536 and word1 u0646 word word2 elif word3 u0627 and word0 u0627 word word1 3 word4 elif word4 u0646 and word3 u0627 word word 3 elif word3 u064a and word0 u062a word word1 3 word4 elif word3 u0648 and word1 u0627 word word0 word2 word4 elif word2 u0627 and word1 u0648 word word0 word3 elif word3 u0626 and word2 u0627 word word 2 word4 elif word4 u0629 and word1 u0627 word word0 word2 4 elif word4 u064a and word2 u0627 word word 2 word3 else word self suf1word do normalize short sufix if lenword 5 word self pre1word do normalize short prefix return word def prow54self word ending step word of length five if lenword 4 word self prow4word elif lenword 5 word self prow54word return word def prow6self word process length six patterns and extract length four roots if word0 u0627 and word4 u0627 word word1 4 word5 elif word startswithu0645u062a word word2 return word def endw6self word normalize short sufix for sf1 in self s1 if word endswithsf1 return word 1 return word def pre1self word natural language toolkit the isri arabic stemmer c 2001 2023 nltk project algorithm kazem taghva rania elkhoury and jeffrey coombs 2005 hosam algasaier hosam_hme yahoo com url https www nltk org for license information see license txt isri arabic stemmer the algorithm for this stemmer is described in taghva k elkoury r and coombs j 2005 arabic stemming without a root dictionary information science research institute university of nevada las vegas usa the information science research institute s isri arabic stemmer shares many features with the khoja stemmer however the main difference is that isri stemmer does not use root dictionary also if a root is not found isri stemmer returned normalized form rather than returning the original unmodified word additional adjustments were made to improve the algorithm 1 adding 60 stop words 2 adding the pattern تفاعيل to isri pattern set 3 the step 2 in the original algorithm was normalizing all hamza this step is discarded because it increases the word ambiguities and changes the original root isri arabic stemmer based on algorithm arabic stemming without a root dictionary information science research institute university of nevada las vegas usa a few minor modifications have been made to isri basic algorithm see the source code of this module for more information isri stem token returns arabic root for the given token the isri stemmer requires that all tokens have unicode string types if you use python idle on arabic windows you have to decode text first using arabic 1256 coding length three prefixes length two prefixes length one prefixes length three suffixes length two suffixes length one suffixes groups of length four patterns groups of length five patterns and length three roots stemming a word token using the isri stemmer remove diacritics which representing arabic short vowels exclude stop words from being processed remove length three and length two prefixes in this order remove length three and length two suffixes in this order remove connective و if it precedes a word beginning with و normalize initial hamza to bare alif if 4 word length 7 then stem otherwise no stemming length 4 word length 5 word length 6 word length 7 word normalization num 1 normalize diacritics num 2 normalize initial hamza num 3 both 1 2 remove length three and length two prefixes in this order remove length three and length two suffixes in this order remove connective و if it precedes a word beginning with و process length four patterns and extract length three roots مفعل فاعل فعال فعول فعيل فعلة do normalize short sufix do normalize short prefix process length five patterns and extract length three roots افتعل افاعل مفعول مفعال مفعيل مفعلة تفعلة افعلة مفتعل يفتعل تفتعل مفاعل تفاعل فعولة فعالة انفعل منفعل افعال فعلان تفعيل فاعول فواعل فعائل فاعلة فعالي do normalize short sufix do normalize short prefix process length five patterns and extract length four roots تفعلل افعلل مفعلل فعللة فعالل ending step word of length five process length six patterns and extract length three roots مستفعل استفعل مفعالة افتعال افعوعل تفاعيل new pattern do normalize short sufix do normalize short prefix process length six patterns and extract length four roots افعلال متفعلل ending step word of length six normalize short sufix normalize short prefix
import re

from nltk.stem.api import StemmerI


class ISRIStemmer(StemmerI):
    

    def __init__(self):
        
        self.p3 = [
            "\u0643\u0627\u0644",
            "\u0628\u0627\u0644",
            "\u0648\u0644\u0644",
            "\u0648\u0627\u0644",
        ]

        
        self.p2 = ["\u0627\u0644", "\u0644\u0644"]

        
        self.p1 = [
            "\u0644",
            "\u0628",
            "\u0641",
            "\u0633",
            "\u0648",
            "\u064a",
            "\u062a",
            "\u0646",
            "\u0627",
        ]

        
        self.s3 = [
            "\u062a\u0645\u0644",
            "\u0647\u0645\u0644",
            "\u062a\u0627\u0646",
            "\u062a\u064a\u0646",
            "\u0643\u0645\u0644",
        ]

        
        self.s2 = [
            "\u0648\u0646",
            "\u0627\u062a",
            "\u0627\u0646",
            "\u064a\u0646",
            "\u062a\u0646",
            "\u0643\u0645",
            "\u0647\u0646",
            "\u0646\u0627",
            "\u064a\u0627",
            "\u0647\u0627",
            "\u062a\u0645",
            "\u0643\u0646",
            "\u0646\u064a",
            "\u0648\u0627",
            "\u0645\u0627",
            "\u0647\u0645",
        ]

        
        self.s1 = ["\u0629", "\u0647", "\u064a", "\u0643", "\u062a", "\u0627", "\u0646"]

        
        self.pr4 = {
            0: ["\u0645"],
            1: ["\u0627"],
            2: ["\u0627", "\u0648", "\u064A"],
            3: ["\u0629"],
        }

        
        self.pr53 = {
            0: ["\u0627", "\u062a"],
            1: ["\u0627", "\u064a", "\u0648"],
            2: ["\u0627", "\u062a", "\u0645"],
            3: ["\u0645", "\u064a", "\u062a"],
            4: ["\u0645", "\u062a"],
            5: ["\u0627", "\u0648"],
            6: ["\u0627", "\u0645"],
        }

        self.re_short_vowels = re.compile(r"[\u064B-\u0652]")
        self.re_hamza = re.compile(r"[\u0621\u0624\u0626]")
        self.re_initial_hamza = re.compile(r"^[\u0622\u0623\u0625]")

        self.stop_words = [
            "\u064a\u0643\u0648\u0646",
            "\u0648\u0644\u064a\u0633",
            "\u0648\u0643\u0627\u0646",
            "\u0643\u0630\u0644\u0643",
            "\u0627\u0644\u062a\u064a",
            "\u0648\u0628\u064a\u0646",
            "\u0639\u0644\u064a\u0647\u0627",
            "\u0645\u0633\u0627\u0621",
            "\u0627\u0644\u0630\u064a",
            "\u0648\u0643\u0627\u0646\u062a",
            "\u0648\u0644\u0643\u0646",
            "\u0648\u0627\u0644\u062a\u064a",
            "\u062a\u0643\u0648\u0646",
            "\u0627\u0644\u064a\u0648\u0645",
            "\u0627\u0644\u0644\u0630\u064a\u0646",
            "\u0639\u0644\u064a\u0647",
            "\u0643\u0627\u0646\u062a",
            "\u0644\u0630\u0644\u0643",
            "\u0623\u0645\u0627\u0645",
            "\u0647\u0646\u0627\u0643",
            "\u0645\u0646\u0647\u0627",
            "\u0645\u0627\u0632\u0627\u0644",
            "\u0644\u0627\u0632\u0627\u0644",
            "\u0644\u0627\u064a\u0632\u0627\u0644",
            "\u0645\u0627\u064a\u0632\u0627\u0644",
            "\u0627\u0635\u0628\u062d",
            "\u0623\u0635\u0628\u062d",
            "\u0623\u0645\u0633\u0649",
            "\u0627\u0645\u0633\u0649",
            "\u0623\u0636\u062d\u0649",
            "\u0627\u0636\u062d\u0649",
            "\u0645\u0627\u0628\u0631\u062d",
            "\u0645\u0627\u0641\u062a\u0626",
            "\u0645\u0627\u0627\u0646\u0641\u0643",
            "\u0644\u0627\u0633\u064a\u0645\u0627",
            "\u0648\u0644\u0627\u064a\u0632\u0627\u0644",
            "\u0627\u0644\u062d\u0627\u0644\u064a",
            "\u0627\u0644\u064a\u0647\u0627",
            "\u0627\u0644\u0630\u064a\u0646",
            "\u0641\u0627\u0646\u0647",
            "\u0648\u0627\u0644\u0630\u064a",
            "\u0648\u0647\u0630\u0627",
            "\u0644\u0647\u0630\u0627",
            "\u0641\u0643\u0627\u0646",
            "\u0633\u062a\u0643\u0648\u0646",
            "\u0627\u0644\u064a\u0647",
            "\u064a\u0645\u0643\u0646",
            "\u0628\u0647\u0630\u0627",
            "\u0627\u0644\u0630\u0649",
        ]

    def stem(self, token):
        
        token = self.norm(
            token, 1
        )  
        if token in self.stop_words:
            return token  
        token = self.pre32(
            token
        )  
        token = self.suf32(
            token
        )  
        token = self.waw(
            token
        )  
        token = self.norm(token, 2)  
        
        if len(token) == 4:  
            token = self.pro_w4(token)
        elif len(token) == 5:  
            token = self.pro_w53(token)
            token = self.end_w5(token)
        elif len(token) == 6:  
            token = self.pro_w6(token)
            token = self.end_w6(token)
        elif len(token) == 7:  
            token = self.suf1(token)
            if len(token) == 7:
                token = self.pre1(token)
            if len(token) == 6:
                token = self.pro_w6(token)
                token = self.end_w6(token)
        return token

    def norm(self, word, num=3):
        
        if num == 1:
            word = self.re_short_vowels.sub("", word)
        elif num == 2:
            word = self.re_initial_hamza.sub("\u0627", word)
        elif num == 3:
            word = self.re_short_vowels.sub("", word)
            word = self.re_initial_hamza.sub("\u0627", word)
        return word

    def pre32(self, word):
        
        if len(word) >= 6:
            for pre3 in self.p3:
                if word.startswith(pre3):
                    return word[3:]
        if len(word) >= 5:
            for pre2 in self.p2:
                if word.startswith(pre2):
                    return word[2:]
        return word

    def suf32(self, word):
        
        if len(word) >= 6:
            for suf3 in self.s3:
                if word.endswith(suf3):
                    return word[:-3]
        if len(word) >= 5:
            for suf2 in self.s2:
                if word.endswith(suf2):
                    return word[:-2]
        return word

    def waw(self, word):
        
        if len(word) >= 4 and word[:2] == "\u0648\u0648":
            word = word[1:]
        return word

    def pro_w4(self, word):
        
        if word[0] in self.pr4[0]:  
            word = word[1:]
        elif word[1] in self.pr4[1]:  
            word = word[:1] + word[2:]
        elif word[2] in self.pr4[2]:  
            word = word[:2] + word[3]
        elif word[3] in self.pr4[3]:  
            word = word[:-1]
        else:
            word = self.suf1(word)  
            if len(word) == 4:
                word = self.pre1(word)  
        return word

    def pro_w53(self, word):
        
        if word[2] in self.pr53[0] and word[0] == "\u0627":  
            word = word[1] + word[3:]
        elif word[3] in self.pr53[1] and word[0] == "\u0645":  
            word = word[1:3] + word[4]
        elif word[0] in self.pr53[2] and word[4] == "\u0629":  
            word = word[1:4]
        elif word[0] in self.pr53[3] and word[2] == "\u062a":  
            word = word[1] + word[3:]
        elif word[0] in self.pr53[4] and word[2] == "\u0627":  
            word = word[1] + word[3:]
        elif word[2] in self.pr53[5] and word[4] == "\u0629":  
            word = word[:2] + word[3]
        elif word[0] in self.pr53[6] and word[1] == "\u0646":  
            word = word[2:]
        elif word[3] == "\u0627" and word[0] == "\u0627":  
            word = word[1:3] + word[4]
        elif word[4] == "\u0646" and word[3] == "\u0627":  
            word = word[:3]
        elif word[3] == "\u064a" and word[0] == "\u062a":  
            word = word[1:3] + word[4]
        elif word[3] == "\u0648" and word[1] == "\u0627":  
            word = word[0] + word[2] + word[4]
        elif word[2] == "\u0627" and word[1] == "\u0648":  
            word = word[0] + word[3:]
        elif word[3] == "\u0626" and word[2] == "\u0627":  
            word = word[:2] + word[4]
        elif word[4] == "\u0629" and word[1] == "\u0627":  
            word = word[0] + word[2:4]
        elif word[4] == "\u064a" and word[2] == "\u0627":  
            word = word[:2] + word[3]
        else:
            word = self.suf1(word)  
            if len(word) == 5:
                word = self.pre1(word)  
        return word

    def pro_w54(self, word):
        
        if word[0] in self.pr53[2]:  
            word = word[1:]
        elif word[4] == "\u0629":  
            word = word[:4]
        elif word[2] == "\u0627":  
            word = word[:2] + word[3:]
        return word

    def end_w5(self, word):
        
        if len(word) == 4:
            word = self.pro_w4(word)
        elif len(word) == 5:
            word = self.pro_w54(word)
        return word

    def pro_w6(self, word):
        
        if word.startswith("\u0627\u0633\u062a") or word.startswith(
            "\u0645\u0633\u062a"
        ):  
            word = word[3:]
        elif (
            word[0] == "\u0645" and word[3] == "\u0627" and word[5] == "\u0629"
        ):  
            word = word[1:3] + word[4]
        elif (
            word[0] == "\u0627" and word[2] == "\u062a" and word[4] == "\u0627"
        ):  
            word = word[1] + word[3] + word[5]
        elif (
            word[0] == "\u0627" and word[3] == "\u0648" and word[2] == word[4]
        ):  
            word = word[1] + word[4:]
        elif (
            word[0] == "\u062a" and word[2] == "\u0627" and word[4] == "\u064a"
        ):  
            word = word[1] + word[3] + word[5]
        else:
            word = self.suf1(word)  
            if len(word) == 6:
                word = self.pre1(word)  
        return word

    def pro_w64(self, word):
        
        if word[0] == "\u0627" and word[4] == "\u0627":  
            word = word[1:4] + word[5]
        elif word.startswith("\u0645\u062a"):  
            word = word[2:]
        return word

    def end_w6(self, word):
        
        if len(word) == 5:
            word = self.pro_w53(word)
            word = self.end_w5(word)
        elif len(word) == 6:
            word = self.pro_w64(word)
        return word

    def suf1(self, word):
        
        for sf1 in self.s1:
            if word.endswith(sf1):
                return word[:-1]
        return word

    def pre1(self, word):
        
        for sp1 in self.p1:
            if word.startswith(sp1):
                return word[1:]
        return word