natural language toolkit arlstem stemmer v2 c 20012023 nltk project kheireddine abainia xprogramer k abainiagmail com algorithms kheireddine abainia k abainiagmail com hamza rebbani hamrebbanigmail com url https www nltk org for license information see license txt arlstem2 arabic light stemmer the details about the implementation of this algorithm are described in k abainia and h rebbani comparing the effectiveness of the improved arlstem algorithm with existing arabic light stemmers international conference on theoretical and applicative aspects of computer science ictaacs 19 skikda algeria december 1516 2019 arlstem2 is an arabic light stemmer based on removing the affixes from the words i e prefixes suffixes and infixes it is an improvement of the previous arabic light stemmer arlstem the new version was compared to the original algorithm and several existing arabic light stemmers where the results showed that the new version considerably improves the understemming errors that are common to light stemmers both arlstem and arlstem2 can be run online and do not use any dictionary return a stemmed arabic word after removing affixes this an improved version of the previous algorithm which reduces understemming errors typically used in arabic search engine information retrieval and nlp from nltk stem import arlstem2 stemmer arlstem2 word stemmer stem printword param token the input arabic word unicode to be stemmed type token unicode return a unicode arabic word different alif with hamza alif laam laam laam fa laam fa ba ba alif laam kaaf alif laam waaw alif laam fa laam laam waaw laam laam fa ba alif laam waaw ba alif laam fa kaaf alif laam kaf yaa kaf miim ha alif ha miim kaf miim alif kaf noon shadda ha miim alif ha noon shadda alif noon ya noon waaw noon taa alif noon taa ya noon alif noon waaw noon siin taa siin yaa siin alif siin noon lam noon lam taa lam yaa lam hamza taa miim alif taa noon shadda noon alif taa miim taa alif waaw alif taa alif noon call this function to get the first stem remove arabic diacritics and replace some letters with others strip the common noun prefixes transform the feminine form to masculine form strip the adjective affixes strip the suffixes that are common to nouns and verbs transform a plural noun to a singular noun strip the verb prefixes and suffixes stem the input word run the first round of stemming check if there is some additional noun affixes taa yaa char miim waaw char alif yaa laam normalize the word by removing diacritics replace hamzated alif with alif bare replace alifmaqsura with yaa and remove waaw at the beginning strip arabic diacritics replace hamzated alif with alif bare replace alifmaqsura with yaa strip the waaw from the word beginning if the remaining is triliteral at least remove prefixes from the words beginning remove the infixes from adjectives alif alif yaa remove the suffixes from the word s ending noon and alif transform the word from the feminine form to the masculine form taa yaa yaa and taa marbuta alif yaa yaa and taa marbuta alif yaa and taa marbuta alif taa marbuta yaa and taa marbuta taa marbuta transform the word from the plural form to the singular form haa noon waaw alif taa alif alif alif alif stem the verb prefixes and suffixes or both stem the present tense cooccurred prefixes and suffixes waaw alif yaa alif noon yaa noon taa noon stem the future tense cooccurred prefixes and suffixes siin taa siin yaa alif noon siin yaa waaw noon siin taa noon siin yaa noon stem the present tense suffixes stem the present tense prefixes stem the future tense prefixes stem the imperative tense prefixes natural language toolkit arlstem stemmer v2 c 2001 2023 nltk project kheireddine abainia x programer k abainia gmail com algorithms kheireddine abainia k abainia gmail com hamza rebbani hamrebbani gmail com url https www nltk org for license information see license txt arlstem2 arabic light stemmer the details about the implementation of this algorithm are described in k abainia and h rebbani comparing the effectiveness of the improved arlstem algorithm with existing arabic light stemmers international conference on theoretical and applicative aspects of computer science ictaacs 19 skikda algeria december 15 16 2019 arlstem2 is an arabic light stemmer based on removing the affixes from the words i e prefixes suffixes and infixes it is an improvement of the previous arabic light stemmer arlstem the new version was compared to the original algorithm and several existing arabic light stemmers where the results showed that the new version considerably improves the under stemming errors that are common to light stemmers both arlstem and arlstem2 can be run online and do not use any dictionary return a stemmed arabic word after removing affixes this an improved version of the previous algorithm which reduces under stemming errors typically used in arabic search engine information retrieval and nlp from nltk stem import arlstem2 stemmer arlstem2 word stemmer stem يعمل print word عمل param token the input arabic word unicode to be stemmed type token unicode return a unicode arabic word different alif with hamza alif laam laam laam fa laam fa ba ba alif laam kaaf alif laam waaw alif laam fa laam laam waaw laam laam fa ba alif laam waaw ba alif laam fa kaaf alif laam kaf yaa kaf miim ha alif ha miim kaf miim alif kaf noon shadda ha miim alif ha noon shadda alif noon ya noon waaw noon taa alif noon taa ya noon alif noon waaw noon siin taa siin yaa siin alif siin noon lam noon lam taa lam yaa lam hamza taa miim alif taa noon shadda noon alif taa miim taa alif waaw alif taa alif noon call this function to get the first stem remove arabic diacritics and replace some letters with others strip the common noun prefixes transform the feminine form to masculine form strip the adjective affixes strip the suffixes that are common to nouns and verbs transform a plural noun to a singular noun if the noun prefixes are not stripped strip the verb prefixes and suffixes stem the input word run the first round of stemming check if there is some additional noun affixes taa yaa char miim waaw char alif yaa laam normalize the word by removing diacritics replace hamzated alif with alif bare replace alifmaqsura with yaa and remove waaw at the beginning strip arabic diacritics replace hamzated alif with alif bare replace alifmaqsura with yaa strip the waaw from the word beginning if the remaining is tri literal at least remove prefixes from the words beginning remove the infixes from adjectives alif alif yaa remove the suffixes from the word s ending noon and alif transform the word from the feminine form to the masculine form taa yaa yaa and taa marbuta alif yaa yaa and taa marbuta alif yaa and taa marbuta alif taa marbuta yaa and taa marbuta taa marbuta transform the word from the plural form to the singular form haa noon waaw alif taa alif alif alif alif stem the verb prefixes and suffixes or both stem the present tense co occurred prefixes and suffixes taa yaa alif waaw alif yaa alif noon yaa noon taa noon stem the future tense co occurred prefixes and suffixes siin taa siin yaa alif noon siin yaa waaw noon siin taa noon siin yaa noon stem the present tense suffixes stem the present tense prefixes stem the future tense prefixes stem the imperative tense prefixes
import re

from nltk.stem.api import StemmerI


class ARLSTem2(StemmerI):
    

    def __init__(self):
        
        self.re_hamzated_alif = re.compile(r"[\u0622\u0623\u0625]")
        self.re_alifMaqsura = re.compile(r"[\u0649]")
        self.re_diacritics = re.compile(r"[\u064B-\u065F]")

        
        self.pr2 = ["\u0627\u0644", "\u0644\u0644", "\u0641\u0644", "\u0641\u0628"]
        
        self.pr3 = ["\u0628\u0627\u0644", "\u0643\u0627\u0644", "\u0648\u0627\u0644"]
        
        self.pr32 = ["\u0641\u0644\u0644", "\u0648\u0644\u0644"]
        
        self.pr4 = [
            "\u0641\u0628\u0627\u0644",
            "\u0648\u0628\u0627\u0644",
            "\u0641\u0643\u0627\u0644",
        ]

        
        self.su2 = ["\u0643\u064A", "\u0643\u0645"]
        
        self.su22 = ["\u0647\u0627", "\u0647\u0645"]
        
        self.su3 = ["\u0643\u0645\u0627", "\u0643\u0646\u0651"]
        
        self.su32 = ["\u0647\u0645\u0627", "\u0647\u0646\u0651"]

        
        self.pl_si2 = ["\u0627\u0646", "\u064A\u0646", "\u0648\u0646"]
        
        self.pl_si3 = ["\u062A\u0627\u0646", "\u062A\u064A\u0646"]

        
        self.verb_su2 = ["\u0627\u0646", "\u0648\u0646"]
        
        self.verb_pr2 = ["\u0633\u062A", "\u0633\u064A"]
        
        self.verb_pr22 = ["\u0633\u0627", "\u0633\u0646"]
        
        self.verb_pr33 = [
            "\u0644\u0646",
            "\u0644\u062A",
            "\u0644\u064A",
            "\u0644\u0623",
        ]
        
        self.verb_suf3 = ["\u062A\u0645\u0627", "\u062A\u0646\u0651"]
        
        self.verb_suf2 = [
            "\u0646\u0627",
            "\u062A\u0645",
            "\u062A\u0627",
            "\u0648\u0627",
        ]
        
        self.verb_suf1 = ["\u062A", "\u0627", "\u0646"]

    def stem1(self, token):
        
        try:
            if token is None:
                raise ValueError(
                    "The word could not be stemmed, because \
                                 it is empty !"
                )
            self.is_verb = False
            
            token = self.norm(token)
            
            pre = self.pref(token)
            if pre is not None:
                token = pre
            
            fm = self.fem2masc(token)
            if fm is not None:
                return fm
            
            adj = self.adjective(token)
            if adj is not None:
                return adj
            
            token = self.suff(token)
            
            ps = self.plur2sing(token)
            if ps is None:
                if pre is None:  
                    
                    verb = self.verb(token)
                    if verb is not None:
                        self.is_verb = True
                        return verb
            else:
                return ps
            return token
        except ValueError as e:
            print(e)

    def stem(self, token):
        
        try:
            if token is None:
                raise ValueError(
                    "The word could not be stemmed, because \
                                 it is empty !"
                )
            
            token = self.stem1(token)
            
            if len(token) > 4:
                
                if token.startswith("\u062A") and token[-2] == "\u064A":
                    token = token[1:-2] + token[-1]
                    return token
                
                if token.startswith("\u0645") and token[-2] == "\u0648":
                    token = token[1:-2] + token[-1]
                    return token
            if len(token) > 3:
                
                if not token.startswith("\u0627") and token.endswith("\u064A"):
                    token = token[:-1]
                    return token
                
                if token.startswith("\u0644"):
                    return token[1:]
            return token
        except ValueError as e:
            print(e)

    def norm(self, token):
        
        
        token = self.re_diacritics.sub("", token)
        
        token = self.re_hamzated_alif.sub("\u0627", token)
        
        token = self.re_alifMaqsura.sub("\u064A", token)
        
        
        if token.startswith("\u0648") and len(token) > 3:
            token = token[1:]
        return token

    def pref(self, token):
        
        if len(token) > 5:
            for p3 in self.pr3:
                if token.startswith(p3):
                    return token[3:]
        if len(token) > 6:
            for p4 in self.pr4:
                if token.startswith(p4):
                    return token[4:]
        if len(token) > 5:
            for p3 in self.pr32:
                if token.startswith(p3):
                    return token[3:]
        if len(token) > 4:
            for p2 in self.pr2:
                if token.startswith(p2):
                    return token[2:]

    def adjective(self, token):
        
        
        if len(token) > 5:
            if (
                token.startswith("\u0627")
                and token[-3] == "\u0627"
                and token.endswith("\u064A")
            ):
                return token[:-3] + token[-2]

    def suff(self, token):
        
        if token.endswith("\u0643") and len(token) > 3:
            return token[:-1]
        if len(token) > 4:
            for s2 in self.su2:
                if token.endswith(s2):
                    return token[:-2]
        if len(token) > 5:
            for s3 in self.su3:
                if token.endswith(s3):
                    return token[:-3]
        if token.endswith("\u0647") and len(token) > 3:
            token = token[:-1]
            return token
        if len(token) > 4:
            for s2 in self.su22:
                if token.endswith(s2):
                    return token[:-2]
        if len(token) > 5:
            for s3 in self.su32:
                if token.endswith(s3):
                    return token[:-3]
        
        if token.endswith("\u0646\u0627") and len(token) > 4:
            return token[:-2]
        return token

    def fem2masc(self, token):
        
        if len(token) > 6:
            
            if (
                token.startswith("\u062A")
                and token[-4] == "\u064A"
                and token.endswith("\u064A\u0629")
            ):
                return token[1:-4] + token[-3]
            
            if (
                token.startswith("\u0627")
                and token[-4] == "\u0627"
                and token.endswith("\u064A\u0629")
            ):
                return token[:-4] + token[-3]
        
        if token.endswith("\u0627\u064A\u0629") and len(token) > 5:
            return token[:-2]
        if len(token) > 4:
            
            if token[1] == "\u0627" and token.endswith("\u0629"):
                return token[0] + token[2:-1]
            
            if token.endswith("\u064A\u0629"):
                return token[:-2]
        
        if token.endswith("\u0629") and len(token) > 3:
            return token[:-1]

    def plur2sing(self, token):
        
        
        if len(token) > 5:
            if token.startswith("\u0645") and token.endswith("\u0648\u0646"):
                return token[1:-2]
        if len(token) > 4:
            for ps2 in self.pl_si2:
                if token.endswith(ps2):
                    return token[:-2]
        if len(token) > 5:
            for ps3 in self.pl_si3:
                if token.endswith(ps3):
                    return token[:-3]
        if len(token) > 4:
            
            if token.endswith("\u0627\u062A"):
                return token[:-2]
            
            if token.startswith("\u0627") and token[2] == "\u0627":
                return token[:2] + token[3:]
            
            if token.startswith("\u0627") and token[-2] == "\u0627":
                return token[1:-2] + token[-1]

    def verb(self, token):
        
        vb = self.verb_t1(token)
        if vb is not None:
            return vb
        vb = self.verb_t2(token)
        if vb is not None:
            return vb
        vb = self.verb_t3(token)
        if vb is not None:
            return vb
        vb = self.verb_t4(token)
        if vb is not None:
            return vb
        vb = self.verb_t5(token)
        if vb is not None:
            return vb
        vb = self.verb_t6(token)
        return vb

    def verb_t1(self, token):
        
        if len(token) > 5 and token.startswith("\u062A"):  
            for s2 in self.pl_si2:
                if token.endswith(s2):
                    return token[1:-2]
        if len(token) > 5 and token.startswith("\u064A"):  
            for s2 in self.verb_su2:
                if token.endswith(s2):
                    return token[1:-2]
        if len(token) > 4 and token.startswith("\u0627"):  
            
            if len(token) > 5 and token.endswith("\u0648\u0627"):
                return token[1:-2]
            
            if token.endswith("\u064A"):
                return token[1:-1]
            
            if token.endswith("\u0627"):
                return token[1:-1]
            
            if token.endswith("\u0646"):
                return token[1:-1]
        
        if len(token) > 4 and token.startswith("\u064A") and token.endswith("\u0646"):
            return token[1:-1]
        
        if len(token) > 4 and token.startswith("\u062A") and token.endswith("\u0646"):
            return token[1:-1]

    def verb_t2(self, token):
        
        if len(token) > 6:
            for s2 in self.pl_si2:
                
                if token.startswith(self.verb_pr2[0]) and token.endswith(s2):
                    return token[2:-2]
            
            if token.startswith(self.verb_pr2[1]) and token.endswith(self.pl_si2[0]):
                return token[2:-2]
            
            if token.startswith(self.verb_pr2[1]) and token.endswith(self.pl_si2[2]):
                return token[2:-2]
        
        if (
            len(token) > 5
            and token.startswith(self.verb_pr2[0])
            and token.endswith("\u0646")
        ):
            return token[2:-1]
        
        if (
            len(token) > 5
            and token.startswith(self.verb_pr2[1])
            and token.endswith("\u0646")
        ):
            return token[2:-1]

    def verb_t3(self, token):
        
        if len(token) > 5:
            for su3 in self.verb_suf3:
                if token.endswith(su3):
                    return token[:-3]
        if len(token) > 4:
            for su2 in self.verb_suf2:
                if token.endswith(su2):
                    return token[:-2]
        if len(token) > 3:
            for su1 in self.verb_suf1:
                if token.endswith(su1):
                    return token[:-1]

    def verb_t4(self, token):
        
        if len(token) > 3:
            for pr1 in self.verb_suf1:
                if token.startswith(pr1):
                    return token[1:]
            if token.startswith("\u064A"):
                return token[1:]

    def verb_t5(self, token):
        
        if len(token) > 4:
            for pr2 in self.verb_pr22:
                if token.startswith(pr2):
                    return token[2:]
            for pr2 in self.verb_pr2:
                if token.startswith(pr2):
                    return token[2:]

    def verb_t6(self, token):
        
        if len(token) > 4:
            for pr3 in self.verb_pr33:
                if token.startswith(pr3):
                    return token[2:]

        return token