natural language toolkit arlstem stemmer c 20012023 nltk project kheireddine abainia xprogramer k abainiagmail com algorithms kheireddine abainia k abainiagmail com siham ouamour halim sayoud url https www nltk org for license information see license txt arlstem arabic stemmer the details about the implementation of this algorithm are described in k abainia s ouamour and h sayoud a novel robust arabic light stemmer journal of experimental theoretical artificial intelligence jetai 17 vol 29 no 3 2017 pp 557573 the arlstem is a light arabic stemmer that is based on removing the affixes from the word i e prefixes suffixes and infixes it was evaluated and compared to several other stemmers using paice s parameters understemming index overstemming index and stemming weight and the results showed that arlstem is promising and producing high performances this stemmer is not based on any dictionary and can be used online effectively arlstem stemmer a light arabic stemming algorithm without any dictionary department of telecommunication information processing usthb university algiers algeria arlstem stemtoken returns the arabic stem for the input token the arlstem stemmer requires that all tokens are encoded using unicode encoding different alif with hamza alif laam laam laam fa laam fa ba ba alif laam kaaf alif laam waaw alif laam fa laam laam waaw laam laam fa ba alif laam waaw ba alif laam fa kaaf alif laam kaf yaa kaf miim ha alif ha miim kaf miim alif kaf noon shadda ha miim alif ha noon shadda alif noon ya noon waaw noon taa alif noon taa ya noon alif noon waaw noon siin taa siin yaa siin alif siin noon lam noon lam taa lam yaa lam hamza taa miim alif taa noon shadda noon alif taa miim taa alif waaw alif taa alif noon call this function to get the word s stem based on arlstem remove arabic diacritics and replace some letters with others strip common prefixes of the nouns strip the suffixes which are common to nouns and verbs transform a plural noun to a singular noun transform from the feminine form to the masculine form strip the verb prefixes and suffixes normalize the word by removing diacritics replacing hamzated alif with alif replacing alifmaqsura with yaa and removing waaw at the beginning strip arabic diacritics replace hamzated alif with alif bare replace alifmaqsura with yaa strip the waaw from the word beginning if the remaining is 3 letters at least remove prefixes from the words beginning remove suffixes from the word s end transform the word from the feminine form to the masculine form transform the word from the plural form to the singular form stem the verb prefixes and suffixes or both stem the present prefixes and suffixes waaw alif yaa alif noon yaa noon taa noon stem the future prefixes and suffixes siin taa siin yaa alif noon siin yaa waaw noon siin taa noon siin yaa noon stem the present suffixes stem the present prefixes stem the future prefixes stem the order prefixes natural language toolkit arlstem stemmer c 2001 2023 nltk project kheireddine abainia x programer k abainia gmail com algorithms kheireddine abainia k abainia gmail com siham ouamour halim sayoud url https www nltk org for license information see license txt arlstem arabic stemmer the details about the implementation of this algorithm are described in k abainia s ouamour and h sayoud a novel robust arabic light stemmer journal of experimental theoretical artificial intelligence jetai 17 vol 29 no 3 2017 pp 557 573 the arlstem is a light arabic stemmer that is based on removing the affixes from the word i e prefixes suffixes and infixes it was evaluated and compared to several other stemmers using paice s parameters under stemming index over stemming index and stemming weight and the results showed that arlstem is promising and producing high performances this stemmer is not based on any dictionary and can be used on line effectively arlstem stemmer a light arabic stemming algorithm without any dictionary department of telecommunication information processing usthb university algiers algeria arlstem stem token returns the arabic stem for the input token the arlstem stemmer requires that all tokens are encoded using unicode encoding different alif with hamza alif laam laam laam fa laam fa ba ba alif laam kaaf alif laam waaw alif laam fa laam laam waaw laam laam fa ba alif laam waaw ba alif laam fa kaaf alif laam kaf yaa kaf miim ha alif ha miim kaf miim alif kaf noon shadda ha miim alif ha noon shadda alif noon ya noon waaw noon taa alif noon taa ya noon alif noon waaw noon siin taa siin yaa siin alif siin noon lam noon lam taa lam yaa lam hamza taa miim alif taa noon shadda noon alif taa miim taa alif waaw alif taa alif noon call this function to get the word s stem based on arlstem remove arabic diacritics and replace some letters with others strip common prefixes of the nouns strip the suffixes which are common to nouns and verbs transform a plural noun to a singular noun transform from the feminine form to the masculine form if the prefixes are not stripped strip the verb prefixes and suffixes normalize the word by removing diacritics replacing hamzated alif with alif replacing alifmaqsura with yaa and removing waaw at the beginning strip arabic diacritics replace hamzated alif with alif bare replace alifmaqsura with yaa strip the waaw from the word beginning if the remaining is 3 letters at least remove prefixes from the words beginning remove suffixes from the word s end transform the word from the feminine form to the masculine form transform the word from the plural form to the singular form stem the verb prefixes and suffixes or both stem the present prefixes and suffixes taa yaa alif waaw alif yaa alif noon yaa noon taa noon stem the future prefixes and suffixes siin taa siin yaa alif noon siin yaa waaw noon siin taa noon siin yaa noon stem the present suffixes stem the present prefixes stem the future prefixes stem the order prefixes
import re

from nltk.stem.api import StemmerI


class ARLSTem(StemmerI):
    

    def __init__(self):
        
        self.re_hamzated_alif = re.compile(r"[\u0622\u0623\u0625]")
        self.re_alifMaqsura = re.compile(r"[\u0649]")
        self.re_diacritics = re.compile(r"[\u064B-\u065F]")

        
        self.pr2 = ["\u0627\u0644", "\u0644\u0644", "\u0641\u0644", "\u0641\u0628"]
        
        self.pr3 = ["\u0628\u0627\u0644", "\u0643\u0627\u0644", "\u0648\u0627\u0644"]
        
        self.pr32 = ["\u0641\u0644\u0644", "\u0648\u0644\u0644"]
        
        self.pr4 = [
            "\u0641\u0628\u0627\u0644",
            "\u0648\u0628\u0627\u0644",
            "\u0641\u0643\u0627\u0644",
        ]

        
        self.su2 = ["\u0643\u064A", "\u0643\u0645"]
        
        self.su22 = ["\u0647\u0627", "\u0647\u0645"]
        
        self.su3 = ["\u0643\u0645\u0627", "\u0643\u0646\u0651"]
        
        self.su32 = ["\u0647\u0645\u0627", "\u0647\u0646\u0651"]

        
        self.pl_si2 = ["\u0627\u0646", "\u064A\u0646", "\u0648\u0646"]
        
        self.pl_si3 = ["\u062A\u0627\u0646", "\u062A\u064A\u0646"]

        
        self.verb_su2 = ["\u0627\u0646", "\u0648\u0646"]
        
        self.verb_pr2 = ["\u0633\u062A", "\u0633\u064A"]
        
        self.verb_pr22 = ["\u0633\u0627", "\u0633\u0646"]
        
        self.verb_pr33 = [
            "\u0644\u0646",
            "\u0644\u062A",
            "\u0644\u064A",
            "\u0644\u0623",
        ]
        
        self.verb_suf3 = ["\u062A\u0645\u0627", "\u062A\u0646\u0651"]
        
        self.verb_suf2 = [
            "\u0646\u0627",
            "\u062A\u0645",
            "\u062A\u0627",
            "\u0648\u0627",
        ]
        
        self.verb_suf1 = ["\u062A", "\u0627", "\u0646"]

    def stem(self, token):
        
        try:
            if token is None:
                raise ValueError(
                    "The word could not be stemmed, because \
                                 it is empty !"
                )
            
            token = self.norm(token)
            
            pre = self.pref(token)
            if pre is not None:
                token = pre
            
            token = self.suff(token)
            
            ps = self.plur2sing(token)
            if ps is None:
                
                fm = self.fem2masc(token)
                if fm is not None:
                    return fm
                else:
                    if pre is None:  
                        
                        return self.verb(token)
            else:
                return ps
            return token
        except ValueError as e:
            print(e)

    def norm(self, token):
        
        
        token = self.re_diacritics.sub("", token)
        
        token = self.re_hamzated_alif.sub("\u0627", token)
        
        token = self.re_alifMaqsura.sub("\u064A", token)
        
        
        if token.startswith("\u0648") and len(token) > 3:
            token = token[1:]
        return token

    def pref(self, token):
        
        if len(token) > 5:
            for p3 in self.pr3:
                if token.startswith(p3):
                    return token[3:]
        if len(token) > 6:
            for p4 in self.pr4:
                if token.startswith(p4):
                    return token[4:]
        if len(token) > 5:
            for p3 in self.pr32:
                if token.startswith(p3):
                    return token[3:]
        if len(token) > 4:
            for p2 in self.pr2:
                if token.startswith(p2):
                    return token[2:]

    def suff(self, token):
        
        if token.endswith("\u0643") and len(token) > 3:
            return token[:-1]
        if len(token) > 4:
            for s2 in self.su2:
                if token.endswith(s2):
                    return token[:-2]
        if len(token) > 5:
            for s3 in self.su3:
                if token.endswith(s3):
                    return token[:-3]
        if token.endswith("\u0647") and len(token) > 3:
            token = token[:-1]
            return token
        if len(token) > 4:
            for s2 in self.su22:
                if token.endswith(s2):
                    return token[:-2]
        if len(token) > 5:
            for s3 in self.su32:
                if token.endswith(s3):
                    return token[:-3]
        if token.endswith("\u0646\u0627") and len(token) > 4:
            return token[:-2]
        return token

    def fem2masc(self, token):
        
        if token.endswith("\u0629") and len(token) > 3:
            return token[:-1]

    def plur2sing(self, token):
        
        if len(token) > 4:
            for ps2 in self.pl_si2:
                if token.endswith(ps2):
                    return token[:-2]
        if len(token) > 5:
            for ps3 in self.pl_si3:
                if token.endswith(ps3):
                    return token[:-3]
        if len(token) > 3 and token.endswith("\u0627\u062A"):
            return token[:-2]
        if len(token) > 3 and token.startswith("\u0627") and token[2] == "\u0627":
            return token[:2] + token[3:]
        if len(token) > 4 and token.startswith("\u0627") and token[-2] == "\u0627":
            return token[1:-2] + token[-1]

    def verb(self, token):
        
        vb = self.verb_t1(token)
        if vb is not None:
            return vb
        vb = self.verb_t2(token)
        if vb is not None:
            return vb
        vb = self.verb_t3(token)
        if vb is not None:
            return vb
        vb = self.verb_t4(token)
        if vb is not None:
            return vb
        vb = self.verb_t5(token)
        if vb is not None:
            return vb
        return self.verb_t6(token)

    def verb_t1(self, token):
        
        if len(token) > 5 and token.startswith("\u062A"):  
            for s2 in self.pl_si2:
                if token.endswith(s2):
                    return token[1:-2]
        if len(token) > 5 and token.startswith("\u064A"):  
            for s2 in self.verb_su2:
                if token.endswith(s2):
                    return token[1:-2]
        if len(token) > 4 and token.startswith("\u0627"):  
            
            if len(token) > 5 and token.endswith("\u0648\u0627"):
                return token[1:-2]
            
            if token.endswith("\u064A"):
                return token[1:-1]
            
            if token.endswith("\u0627"):
                return token[1:-1]
            
            if token.endswith("\u0646"):
                return token[1:-1]
        
        if len(token) > 4 and token.startswith("\u064A") and token.endswith("\u0646"):
            return token[1:-1]
        
        if len(token) > 4 and token.startswith("\u062A") and token.endswith("\u0646"):
            return token[1:-1]

    def verb_t2(self, token):
        
        if len(token) > 6:
            for s2 in self.pl_si2:
                
                if token.startswith(self.verb_pr2[0]) and token.endswith(s2):
                    return token[2:-2]
            
            if token.startswith(self.verb_pr2[1]) and token.endswith(self.pl_si2[0]):
                return token[2:-2]
            
            if token.startswith(self.verb_pr2[1]) and token.endswith(self.pl_si2[2]):
                return token[2:-2]
        
        if (
            len(token) > 5
            and token.startswith(self.verb_pr2[0])
            and token.endswith("\u0646")
        ):
            return token[2:-1]
        
        if (
            len(token) > 5
            and token.startswith(self.verb_pr2[1])
            and token.endswith("\u0646")
        ):
            return token[2:-1]

    def verb_t3(self, token):
        
        if len(token) > 5:
            for su3 in self.verb_suf3:
                if token.endswith(su3):
                    return token[:-3]
        if len(token) > 4:
            for su2 in self.verb_suf2:
                if token.endswith(su2):
                    return token[:-2]
        if len(token) > 3:
            for su1 in self.verb_suf1:
                if token.endswith(su1):
                    return token[:-1]

    def verb_t4(self, token):
        
        if len(token) > 3:
            for pr1 in self.verb_suf1:
                if token.startswith(pr1):
                    return token[1:]
            if token.startswith("\u064A"):
                return token[1:]

    def verb_t5(self, token):
        
        if len(token) > 4:
            for pr2 in self.verb_pr22:
                if token.startswith(pr2):
                    return token[2:]
            for pr2 in self.verb_pr2:
                if token.startswith(pr2):
                    return token[2:]
        return token

    def verb_t6(self, token):
        
        if len(token) > 4:
            for pr3 in self.verb_pr33:
                if token.startswith(pr3):
                    return token[2:]
        return token