<jupyter_start><jupyter_text>Lemmatization <NAME> (DTU/2K16/MC/013) Natural Language Processing (NLP) - Dr. <NAME> OverviewWe introduce lemmatization in this Notebook and alsous lemmatization to process our resume to lemmatize each toke i the corpus (Resume). We also compare the results with previously seen Porter Stemmer algorithm on our Resume. 1. IntroductionFor grammatical reasons, documents are going to use different forms of a word, such as organize, organizes, and organizing. Additionally, there are families of derivationally related words with similar meanings, such as democracy, democratic, and democratization. In many situations, it seems as if it would be useful for a search for one of these words to return documents that contain another word in the set.The goal of both stemming and lemmatization is to reduce inflectional forms and sometimes derivationally related forms of a word to a common base form. For instance:_am, are, is_ $\Rightarrow$ __be___car, cars, car's, cars'_ $\Rightarrow$ __car__The Result of such mappings can b something like this:_the boy's cars are different colors_ $\Rightarrow$ _the boy car be differ color_However, the two words differ in their flavor. Stemming usually refers to a crude heuristic process that chops off the ends of words in the hope of achieving this goal correctly most of the time, and often includes the removal of derivational affixes. Lemmatization usually refers to doing things properly with the use of a vocabulary and morphological analysis of words, normally aiming to remove inflectional endings only and to return the base or dictionary form of a word, which is known as the lemma . If confronted with the token _saw_, stemming might return just _s_, whereas lemmatization would attempt to return either _see_ or _saw_ depending on whether the use of the token was as a verb or a noun. The two may also differ in that stemming most commonly collapses derivationally related words, whereas lemmatization commonly only collapses the different inflectional forms of a lemma. Linguistic processing for stemming or lemmatization is often done by an additional plug-in component to the indexing process, and a number of such components exist, both commercial and open-source.  2. Importing basic tools required for tokenization, stemming and lemmatization of the resume corpus.<jupyter_code>import nltk
import pickle
from collections import Counter
from nltk.stem import WordNetLemmatizer
from nltk.tokenize import word_tokenize
from nltk.corpus import wordnet
nltk.download('wordnet')<jupyter_output>[nltk_data] Downloading package wordnet to
[nltk_data]     C:\Users\anish\AppData\Roaming\nltk_data...
[nltk_data]   Package wordnet is already up-to-date!<jupyter_text>3. TokenizerThe Tokenizer is a basic tool used in NLP that returns tokens from a given document. We can create our own token as well based on Regex.<jupyter_code>class Tokenizer:
    def __init__(self):
        self._tokenizer = nltk.RegexpTokenizer(r'\w+')

    def tokenize(self, document: str) -> list:
        return self._tokenizer.tokenize(document)
# We define a basic tokenizer that takes all word strings from a document as a token. Test it below
# You can also modify the sentence below to see tokens
message = "Have no fear of perfection. You'll never reach it 🔥"
tokenizer = Tokenizer()
print(tokenizer.tokenize(message))<jupyter_output>['Have', 'no', 'fear', 'of', 'perfection', 'You', 'll', 'never', 'reach', 'it']<jupyter_text>4. Porter StemmerWe now define the Porter Stemmer Algorithm, one of the most famous stemming algorithm for the english language created by <NAME> in 1980. This algorithm reduces a token by removing it's inflections. e.g. _running_ $\rightarrow$ _run_.We use the porter stemmer as defined in the first assignment which can be seen [here](https://github.com/anishLearnsToCode/porter-stemmer).<jupyter_code>class PorterStemmer:

    def __init__(self):
        """The word is a buffer holding a word to be stemmed. The letters are in the range
        [start, offset ... offset + 1) ... ending at end."""

        self.vowels = ('a', 'e', 'i', 'o', 'u')
        self.word = ''
        self.end = 0
        self.start = 0
        self.offset = 0
        self._tokenizer = Tokenizer()

    def is_vowel(self, letter):
        return letter in self.vowels

    def is_consonant(self, index):
        """:returns True if word[index] is a consonant."""
        if self.is_vowel(self.word[index]):
            return False
        if self.word[index] == 'y':
            if index == self.start:
                return True
            else:
                return not self.is_consonant(index - 1)
        return True

    def m(self):
        """m() measures the number of consonant sequences between start and offset.
        if c is a consonant sequence and v a vowel sequence, and <..>
        indicates arbitrary presence,
           <c><v>       gives 0
           <c>vc<v>     gives 1
           <c>vcvc<v>   gives 2
           <c>vcvcvc<v> gives 3
           ....
        """
        n = 0
        i = self.start
        while True:
            if i > self.offset:
                return n
            if not self.is_consonant(i):
                break
            i += 1
        i += 1
        while True:
            while True:
                if i > self.offset:
                    return n
                if self.is_consonant(i):
                    break
                i += 1
            i += 1
            n += 1
            while True:
                if i > self.offset:
                    return n
                if not self.is_consonant(i):
                    break
                i += 1
            i += 1

    def contains_vowel(self):
        """:returns TRUE if the word contains a vowel in the range [start, offset]"""
        for i in range(self.start, self.offset + 1):
            if not self.is_consonant(i):
                return True
        return False

    def contains_double_consonant(self, j):
        """:returns TRUE if the word contain a double consonant in the range [offset, start]"""
        if j < (self.start + 1):
            return False
        if self.word[j] != self.word[j - 1]:
            return False
        return self.is_consonant(j)

    def is_of_form_cvc(self, i):
        """:returns TRUE for indices set {i-2, i-1, i} has the form consonant - vowel - consonant
        and also if the second c is not w,x or y. this is used when trying to
        restore an e at the end of a short  e.g.
           cav(e), lov(e), hop(e), crim(e), but
           snow, box, tray.
        """
        if i < (self.start + 2) or not self.is_consonant(i) or self.is_consonant(i - 1) or not self.is_consonant(i - 2):
            return 0
        ch = self.word[i]
        if ch == 'w' or ch == 'x' or ch == 'y':
            return 0
        return 1

    def ends_with(self, s):
        """:returns TRUE when {start...end} ends with the string s."""
        length = len(s)
        if s[length - 1] != self.word[self.end]:  # tiny speed-up
            return False
        if length > (self.end - self.start + 1):
            return False
        if self.word[self.end - length + 1: self.end + 1] != s:
            return False
        self.offset = self.end - length
        return True

    def set_to(self, s):
        """sets [offset + 1, end] to the characters in the string s, readjusting end."""
        length = len(s)
        self.word = self.word[:self.offset + 1] + s + self.word[self.offset + length + 1:]
        self.end = self.offset + length

    def replace_morpheme(self, s):
        """is a mapping function to change morphemes"""
        if self.m() > 0:
            self.set_to(s)

    def remove_plurals(self):
        """This is step 1 ab and gets rid of plurals and -ed or -ing. e.g.
           caresses  ->  caress
           ponies    ->  poni
           ties      ->  ti
           caress    ->  caress
           cats      ->  cat
           feed      ->  feed
           agreed    ->  agree
           disabled  ->  disable
           matting   ->  mat
           mating    ->  mate
           meeting   ->  meet
           milling   ->  mill
           messing   ->  mess
           meetings  ->  meet
        """
        if self.word[self.end] == 's':
            if self.ends_with("sses"):
                self.end = self.end - 2
            elif self.ends_with("ies"):
                self.set_to("i")
            elif self.word[self.end - 1] != 's':
                self.end = self.end - 1
        if self.ends_with("eed"):
            if self.m() > 0:
                self.end = self.end - 1
        elif (self.ends_with("ed") or self.ends_with("ing")) and self.contains_vowel():
            self.end = self.offset
            if self.ends_with("at"):
                self.set_to("ate")
            elif self.ends_with("bl"):
                self.set_to("ble")
            elif self.ends_with("iz"):
                self.set_to("ize")
            elif self.contains_double_consonant(self.end):
                self.end = self.end - 1
                ch = self.word[self.end]
                if ch == 'l' or ch == 's' or ch == 'z':
                    self.end = self.end + 1
            elif self.m() == 1 and self.is_of_form_cvc(self.end):
                self.set_to("e")

    def terminal_y_to_i(self):
        """This defines step 1 c which turns terminal y to i when there is another vowel in the stem."""
        if self.ends_with('y') and self.contains_vowel():
            self.word = self.word[:self.end] + 'i' + self.word[self.end + 1:]

    def map_double_to_single_suffix(self):
        """Defines step 2 and maps double suffices to single ones.
        so -ization ( = -ize plus -ation) maps to -ize etc. note that the
        string before the suffix must give m() > 0.
        """
        if self.word[self.end - 1] == 'a':
            if self.ends_with("ational"):
                self.replace_morpheme("ate")
            elif self.ends_with("tional"):
                self.replace_morpheme("tion")
        elif self.word[self.end - 1] == 'c':
            if self.ends_with("enci"):
                self.replace_morpheme("ence")
            elif self.ends_with("anci"):
                self.replace_morpheme("ance")
        elif self.word[self.end - 1] == 'e':
            if self.ends_with("izer"):      self.replace_morpheme("ize")
        elif self.word[self.end - 1] == 'l':
            if self.ends_with("bli"):
                self.replace_morpheme("ble")  # --DEPARTURE--
            # To match the published algorithm, replace this phrase with
            #   if self.ends("abli"):      self.r("able")
            elif self.ends_with("alli"):
                self.replace_morpheme("al")
            elif self.ends_with("entli"):
                self.replace_morpheme("ent")
            elif self.ends_with("eli"):
                self.replace_morpheme("e")
            elif self.ends_with("ousli"):
                self.replace_morpheme("ous")
        elif self.word[self.end - 1] == 'o':
            if self.ends_with("ization"):
                self.replace_morpheme("ize")
            elif self.ends_with("ation"):
                self.replace_morpheme("ate")
            elif self.ends_with("ator"):
                self.replace_morpheme("ate")
        elif self.word[self.end - 1] == 's':
            if self.ends_with("alism"):
                self.replace_morpheme("al")
            elif self.ends_with("iveness"):
                self.replace_morpheme("ive")
            elif self.ends_with("fulness"):
                self.replace_morpheme("ful")
            elif self.ends_with("ousness"):
                self.replace_morpheme("ous")
        elif self.word[self.end - 1] == 't':
            if self.ends_with("aliti"):
                self.replace_morpheme("al")
            elif self.ends_with("iviti"):
                self.replace_morpheme("ive")
            elif self.ends_with("biliti"):
                self.replace_morpheme("ble")
        elif self.word[self.end - 1] == 'g':
            if self.ends_with("logi"):      self.replace_morpheme("log")

    def step3(self):
        """step3() deals with -ic-, -full, -ness etc."""
        if self.word[self.end] == 'e':
            if self.ends_with("icate"):
                self.replace_morpheme("ic")
            elif self.ends_with("ative"):
                self.replace_morpheme("")
            elif self.ends_with("alize"):
                self.replace_morpheme("al")
        elif self.word[self.end] == 'i':
            if self.ends_with("iciti"):     self.replace_morpheme("ic")
        elif self.word[self.end] == 'l':
            if self.ends_with("ical"):
                self.replace_morpheme("ic")
            elif self.ends_with("ful"):
                self.replace_morpheme("")
        elif self.word[self.end] == 's':
            if self.ends_with("ness"):      self.replace_morpheme("")

    def step4(self):
        """step4() takes off -ant, -ence etc., in context <c>vcvc<v>."""
        if self.word[self.end - 1] == 'a':
            if self.ends_with("al"):
                pass
            else:
                return
        elif self.word[self.end - 1] == 'c':
            if self.ends_with("ance"):
                pass
            elif self.ends_with("ence"):
                pass
            else:
                return
        elif self.word[self.end - 1] == 'e':
            if self.ends_with("er"):
                pass
            else:
                return
        elif self.word[self.end - 1] == 'i':
            if self.ends_with("ic"):
                pass
            else:
                return
        elif self.word[self.end - 1] == 'l':
            if self.ends_with("able"):
                pass
            elif self.ends_with("ible"):
                pass
            else:
                return
        elif self.word[self.end - 1] == 'n':
            if self.ends_with("ant"):
                pass
            elif self.ends_with("ement"):
                pass
            elif self.ends_with("ment"):
                pass
            elif self.ends_with("ent"):
                pass
            else:
                return
        elif self.word[self.end - 1] == 'o':
            if self.ends_with("ion") and (self.word[self.offset] == 's' or self.word[self.offset] == 't'):
                pass
            elif self.ends_with("ou"):
                pass
            # takes care of -ous
            else:
                return
        elif self.word[self.end - 1] == 's':
            if self.ends_with("ism"):
                pass
            else:
                return
        elif self.word[self.end - 1] == 't':
            if self.ends_with("ate"):
                pass
            elif self.ends_with("iti"):
                pass
            else:
                return
        elif self.word[self.end - 1] == 'u':
            if self.ends_with("ous"):
                pass
            else:
                return
        elif self.word[self.end - 1] == 'v':
            if self.ends_with("ive"):
                pass
            else:
                return
        elif self.word[self.end - 1] == 'z':
            if self.ends_with("ize"):
                pass
            else:
                return
        else:
            return
        if self.m() > 1:
            self.end = self.offset

    def step5(self):
        """step5() removes a final -e if m() > 1, and changes -ll to -l if m > 1."""
        self.offset = self.end
        if self.word[self.end] == 'e':
            a = self.m()
            if a > 1 or (a == 1 and not self.is_of_form_cvc(self.end - 1)):
                self.end = self.end - 1
        if self.word[self.end] == 'l' and self.contains_double_consonant(self.end) and self.m() > 1:
            self.end = self.end - 1

    def stem_document(self, document):
        result = []
        for line in document.split('\n'):
            result.append(self.stem_sentence(line))
        return '\n'.join(result)

    def alphabetic(self, word):
        return ''.join([letter if letter.isalpha() else '' for letter in word])

    def stem_sentence(self, sentence):
        result = []
        for word in self._tokenizer.tokenize(sentence):
            result.append(self.stem_word(word))
        return ' '.join(result)

    def stem_word(self, word):
        if word == '':
            return ''

        self.word = word
        self.end = len(word) - 1
        self.start = 0

        self.remove_plurals()
        self.terminal_y_to_i()
        self.map_double_to_single_suffix()
        self.step3()
        self.step4()
        self.step5()
        return self.word[self.start: self.end + 1]
# test the porter stemmer with any word of your choice
stemmer = PorterStemmer()
print(stemmer.stem_word('running'))
# test the porter stemmer with any sentence of your choice
print(stemmer.stem_sentence("the boy's cars are different colors"))<jupyter_output>the boi s car ar differ color<jupyter_text>5. LemmatizationWe now create a class for lemmatization that will use the `Tokenizer` class. An instance of the Lemmatization class will help us reduce words into their __lemmas__.<jupyter_code>class Lemmatizer:
    def __init__(self):
        self._lemmatizer = WordNetLemmatizer()
        self._tokenizer = Tokenizer()

    def _tokenize(self, document: str) -> list:
        return self._tokenizer.tokenize(document)

    def lemmatize_word(self, word: str, pos=None) -> str:
        return self._lemmatizer.lemmatize(word, pos) if pos is not None else self._lemmatizer.lemmatize(word)

    def lemmatize_sentence(self, sentence: str, pos=None) -> str:
        result = []
        for word in self._tokenize(sentence):
            if pos is not None:
                result.append(self.lemmatize_word(word, pos))
            else:
                result.append(self.lemmatize_word(word))
        return ' '.join(result)

    def lemmatize_document(self, document: str) -> str:
        result = []
        for line in document.split('\n'):
            result.append(self.lemmatize_sentence(line))
        return '\n'.join(result)
# test the lemmatizer with any word of your choice
lemmatizer = Lemmatizer()
print(lemmatizer.lemmatize_word('wolves'))
# try the lemmaztization algorithm with a sentence of your choice
print(lemmatizer.lemmatize_sentence('the quick brown fox 🦊 jumps over the lazy dog 🐶'))<jupyter_output>the quick brown fox jump over the lazy dog<jupyter_text>6. Loading in the ResumeWe now load in our resume and see the sample text in it.<jupyter_code>resume_file = open('../assets/resume.txt', 'r')
resume = resume_file.read().lower()
resume_file.close()
print(resume)<jupyter_output><NAME>
software developer + clean code enthusiast

phone : 8287428181
email : <EMAIL>
home : sandesh vihar, pitampura, new delhi - 110034
date of birth : 7th april 1998
languages : english, hindi, french

work experience
what after college ( 4 months )
delhi, india
creating content to teach core java and python with data structures and algorithms and giving online classes to students

summer research fellow at university of auckland ( 2 months )
auckland, new zealand
worked on geometry of mobius transformations, differential geometry under dr. pedram hekmati at the department of
mathematics, university of auckland

software developer at cern ( 14 months )
cern, geneva, switzerland
worked in the core platforms team of the fap-bc group. part of an agile team of developers that maintains and adds core
functionality to applications used internally at cern by hr, financial, administrative and other departments
including scientific

worked on legacy applications that compr[...]<jupyter_text>7. Stemming the ResumeWe now Stem our resume by applying the PorterStemmer algorithm on it and see the output.<jupyter_code>stemmer = PorterStemmer()
resume_file = open('../assets/resume.txt', 'r')
resume = resume_file.read().lower()
resume_file.close()

resume_stemmed = stemmer.stem_document(resume)
pickle.dump(obj=resume_stemmed, file=open('../assets/resume_stemmed.p', 'wb'))

resume_stemmed_file = open('../assets/resume_stemmed.txt', 'w')
resume_stemmed_file.write(resume_stemmed)
resume_stemmed_file.close()
# We now display the stemmed resume
print(resume_stemmed)<jupyter_output>anish sachdeva
softwar develop clean code enthusiast

phone 8287428181
email anish_ outlook com
home sandesh vihar pitampura new delhi 110034
date of birth 7th april 1998
languag english hindi french

work experi
what after colleg 4 month
delhi india
creat content to teach core java and python with data structur and algorithm and give onlin class to student

summer research fellow at univers of auckland 2 month
auckland new zealand
work on geometri of mobiu transform differenti geometri under dr pedram hekmati at the depart of
mathemat univers of auckland

softwar develop at cern 14 month
cern geneva switzerland
work in the core platform team of the fap bc group part of an agil team of develop that maintain and add core
function to applic us intern at cern by hr financi administr and other depart
includ scientif

work on legaci applic that compris of singl and some time multipl framework such a java spring boot
hibern and java ee also work with googl polym 1 0 and jsp on the client sid[...]<jupyter_text>8. Creating the Lemmatized ResumeWe now use our `Lemmatizer` class to lemmatize our resume and save it so that we can run analytics on it later on.<jupyter_code>lemmatizer = Lemmatizer()
resume_file = open('../assets/resume.txt')
resume = resume_file.read().lower()
resume_file.close()

resume_lemmatized = lemmatizer.lemmatize_document(resume)
pickle.dump(resume_lemmatized, open('../assets/resume_lemmatized.p', 'wb'))

resume_lemmatized_file = open('../assets/resume_lemmatized.txt', 'w')
resume_lemmatized_file.write(resume_lemmatized)
resume_lemmatized_file.close()
# displaying lemmatized resume
print(resume_lemmatized)<jupyter_output>anish sachdeva
software developer clean code enthusiast

phone 8287428181
email anish_ outlook com
home sandesh vihar pitampura new delhi 110034
date of birth 7th april 1998
language english hindi french

work experience
what after college 4 month
delhi india
creating content to teach core java and python with data structure and algorithm and giving online class to student

summer research fellow at university of auckland 2 month
auckland new zealand
worked on geometry of mobius transformation differential geometry under dr pedram hekmati at the department of
mathematics university of auckland

software developer at cern 14 month
cern geneva switzerland
worked in the core platform team of the fap bc group part of an agile team of developer that maintains and add core
functionality to application used internally at cern by hr financial administrative and other department
including scientific

worked on legacy application that comprise of single and some time multiple framework such a ja[...]<jupyter_text>9. AnalyticsWe now run a few basic analytics and compae the output of the stemmed and lemmaztized Resumes with each other and the original resume.<jupyter_code># We load in the original, stemmed and lemmatized resumes
resume_file = open('../assets/resume.txt', 'r')
resume = resume_file.read().lower()
resume_file.close()
resume_stemmed = pickle.load(open('../assets/resume_stemmed.p', 'rb'))
resume_lemmatized = pickle.load(open('../assets/resume_lemmatized.p', 'rb'))
# extracting tokens from the original, stemmed and lemmatized outputs
resume_tokens = word_tokenize(resume)
stemmed_resume_tokens = word_tokenize(resume_stemmed)
lemmatized_resume_tokens = word_tokenize(resume_lemmatized)
# Comparing the number of tokens in original, stemmed and lemmatized outputs
print('No. of tokens in Resume:', len(resume_tokens))
print('No. of tokens in Stemmed Resume:', len(stemmed_resume_tokens))
print('No. of tokens in Lemmatized Resume:', len(lemmatized_resume_tokens))<jupyter_output>No. of tokens in Resume: 424
No. of tokens in Stemmed Resume: 363
No. of tokens in Lemmatized Resume: 363<jupyter_text>We observe that both the stemmed and lemmatized resume's have same number of tokesn which is correct as the tokenization step for both these processes uses the same Tokenization algorithm.<jupyter_code># comparing no. of words and word frequencies in both stemmed and lemmatized outputs
stemmed_resume_frequencies = Counter(stemmed_resume_tokens)
lemmatized_resume_frequencies = Counter(lemmatized_resume_tokens)
print('\nNo. of unique tokens/words in the stemmed output:', len(stemmed_resume_frequencies))
print('No. of unique tokens/words in the lemmatized output:', len(lemmatized_resume_frequencies))<jupyter_output>No. of unique tokens/words in the stemmed output: 219
No. of unique tokens/words in the lemmatized output: 228<jupyter_text>In the stemmed output there are less number of tokens, but the reduction in number of tokens isn't that high and if the purpose of our task is to reduce the number of tokens in the corpus, stemming is definately a way to go, but lemmatization also achieves similar percentage reduction.<jupyter_code># seeing the top 30 most common words in the stemmed and lemmatized outputs
print('\nTop 30 most common words/tokens in the stemmed output:\n', stemmed_resume_frequencies.most_common(30))
print('\nTop 30 most common words/tokens in the lemmatized output:\n', lemmatized_resume_frequencies.most_common(30))<jupyter_output>Top 30 most common words/tokens in the stemmed output:
 [('and', 16), ('of', 12), ('work', 7), ('java', 7), ('the', 6), ('with', 5), ('on', 5), ('develop', 4), ('com', 4), ('delhi', 4), ('month', 4), ('to', 4), ('core', 4), ('data', 4), ('structur', 4), ('algorithm', 4), ('at', 4), ('univers', 4), ('cern', 4), ('in', 4), ('comput', 4), ('code', 3), ('creat', 3), ('teach', 3), ('auckland', 3), ('mathemat', 3), ('that', 3), ('applic', 3), ('http', 3), ('softwar', 2)]

Top 30 most common words/tokens in the lemmatized output:
 [('and', 16), ('of', 12), ('java', 7), ('worked', 6), ('the', 6), ('with', 5), ('on', 5), ('com', 4), ('delhi', 4), ('month', 4), ('to', 4), ('core', 4), ('data', 4), ('structure', 4), ('algorithm', 4), ('at', 4), ('university', 4), ('cern', 4), ('in', 4), ('developer', 3), ('auckland', 3), ('mathematics', 3), ('that', 3), ('application', 3), ('computer', 3), ('http', 3), ('software', 2), ('new', 2), ('english', 2), ('4', 2)]<jupyter_text>In stemming we are getting words reduced down to their roots and words are much more clearer in their meaning and are closer to their original form in the lemmatizaed format. Although in lemmatization we are also receiving a number __4__ in our most frequently ocurring characters.We now introduce a helpee method that will help us in tagging each token in the corpus with the corresponding Part of Speech Tags (POS tags). POS Tags are mainly of the following types:- Noun (n)- Verb (v)- Adjective (a)- Adverb (r)- Symbol (s)By tagging the orignal, stemmed and lemmatized resumes we can check whether we are still maintaing the same frequency of POS tags. Which will further show that the meaning or context of our words has been retained despite the pre-processing steps of Stemming or Lemmatization.<jupyter_code>def get_pos_frequency(tokens: list) -> Counter:
    synsets = [wordnet.synsets(token) for token in tokens]
    pos_tags = []
    for synset in synsets:
        if isinstance(synset, list) and len(synset) > 0:
            pos_tags.append(synset[0].pos())
    return Counter(pos_tags)


# Analyzing of frequency of POS tags in original, stemmed and Lemmatized resume
resume_pos_frequency = get_pos_frequency(resume_tokens)
stemmed_resume_pos_frequency = get_pos_frequency(stemmed_resume_tokens)
lemmatized_resume_pos_frequency = get_pos_frequency(lemmatized_resume_tokens)

print('\nResume POS Tags Frequency:', resume_pos_frequency)
print('Stemmed Resume POS Tags Frequency:', stemmed_resume_pos_frequency)
print('Lemmatized Resume POS Tags Frequency:', lemmatized_resume_pos_frequency)<jupyter_output>Resume POS Tags Frequency: Counter({'n': 203, 'v': 20, 'a': 18, 's': 10, 'r': 4})
Stemmed Resume POS Tags Frequency: Counter({'n': 161, 'a': 11, 'v': 10, 's': 7, 'r': 4})
Lemmatized Resume POS Tags Frequency: Counter({'n': 215, 'v': 20, 'a': 18, 's': 11, 'r': 5})<jupyter_text>We observe that the number of nouns and adverbs increases in the resume after performing the lemmatization step and the number of nouns clearly decreases after performing stemming. So, if in our application the user wishes to search proper nouns and obtain specific results and only those that match exactly, like searching __java__, __python__ etc. through a resume, the lemmatization will give a better result.We now see the POS (Part of speech) Tags for those words that have been reduced differently by the stemming and Lemmatization Operations.<jupyter_code># compiling pos tags for words that have been stemmed and lemmatized differently
import pprint


printer = pprint.PrettyPrinter(width=50)
diff = []

for stemmed, lemmatized in zip(stemmed_resume_tokens, lemmatized_resume_tokens):
    if not stemmed == lemmatized:
        stemmed_synsets = wordnet.synsets(stemmed)
        lemmatized_synsets = wordnet.synsets(lemmatized)
        stemmed_synset = stemmed_synsets[0].pos() if isinstance(stemmed_synsets, list) and len(stemmed_synsets) > 0 else ''
        lemmatized_synset = lemmatized_synsets[0].pos() if isinstance(lemmatized_synsets, list) and len(lemmatized_synsets) > 0 else ''
        diff.append({stemmed: stemmed_synset, lemmatized: lemmatized_synset})
print('POS Tags for Words with different stemmed and Lemmatized forms:')
printer.pprint(diff)<jupyter_output>POS Tags for Words with different stemmed and Lemmatized forms:
[{'softwar': '', 'software': 'n'},
 {'develop': 'v', 'developer': 'n'},
 {'languag': '', 'language': 'n'},
 {'experi': '', 'experience': 'n'},
 {'colleg': '', 'college': 'n'},
 {'creat': '', 'creating': 'v'},
 {'structur': '', 'structure': 'n'},
 {'give': 'n', 'giving': 'n'},
 {'onlin': '', 'online': 'a'},
 {'univers': '', 'university': 'n'},
 {'work': 'n', 'worked': 'v'},
 {'geometri': '', 'geometry': 'n'},
 {'mobiu': '', 'mobius': 'n'},
 {'transform': 'v', 'transformation': 'n'},
 {'differenti': '', 'differential': 'n'},
 {'geometri': '', 'geometry': 'n'},
 {'depart': 'v', 'department': 'n'},
 {'mathemat': '', 'mathematics': 'n'},
 {'univers': '', 'university': 'n'},
 {'softwar': '', 'software': 'n'},
 {'develop': 'v', 'developer': 'n'},
 {'work': 'n', 'worked': 'v'},
 {'agil': '', 'agile': 's'},
 {'develop': 'v', 'developer': 'n'},
 {'maintain': 'v', 'maintains': 'v'},
 {'function': 'n', 'functionality': 'n'},
 {'applic[...]<jupyter_text>From the above example we can clealy see that teh POS tager present inside the NLTKpackage is unable to tag email ID's, phone numbers, proper nouns such as names etc. for both the _stemmed_ and _lemmatized_ outputs, but it is able to tag ore words in the _lemmatized_ tokens than it is for _stemmed_ words.This is because in lemmatiation we are reducing the words to valid english words and the POS tagger can still tag it as a Noun, verb etc. wheras in the stemmer in many cases the words are reduced to a form that may be benificial for certain NLP tasks like searching, information extraction etc. but the reduced isn't always a valid english word e.g. _pony_ $\rightarrow$ _poni_ ,  _happy_ $\rightarrow$ _happi_ , _language_ $\rightarrow$ _languag_.Hence if we need to use an application where speech tagging is required such as text 2 speech or understanding speech and analyzing speech such as speech to text or machine transaltion Lemmatization can play a vital role wheras Stemming has no percievable bennefits and on the contrary might only reduce application performance.We can also see in afew examples that stemming has actually resulted in a loss of the contextual meaning of the wrd e.g. in the  word _worked_. This word is stemmed down to _work_ and lemmatized down to _worked_. These are then tagged as the follows:1. (Stemming) _worked_ $\rightarrow$ __Noun__1. (Lemmatization) _worked_ $\rightarrow$ __Verb__In the context of our resume, when we wrote worked, we meant it as a verb and not a noun and that context has been lost in stemming. We can see similar results for _learning_ , _development_ and electronic_.  10. Discussion & ConclusionBoth stemming and Lemmatization serve a very similar purpose in the NLP preprocessing pipeline and are methods used to reduce a word to it's toot form, they also have many differences.Stemming reduces a word by removing it's inflections and the morphologically reduced form that it recieives may or may not be an english language word. Stemming uses a heuristic approach to reducing inflections wherein the steps followed are not aware of the meaning of the word or even the context of the word and simply follow basic deterministic rules based on the characters in the word to add and remove character strings in the word.This isn't how lemmatization functions. The Porter Stemmer Algorithm was introduced in the 1980's whereas Lemmatization is a much more modern algorithm that isn't a simple heuristic based algorithm, but it is aware of the word meaning along with synonyms of the word.It also tags the word with the correct part of speech (POS) tag, or multiple tags and uses a pre-built dictionary to define the meaning of each and every tag with different contexts (Contexts being Noun, Verb, Adverb etc.) Lemmatization requires this pre-built dictionary to function and in the __nltk__ package that we were using, internally __nltk__ uses the __wordnet__ corpus which contains words, their definitions for all different contexts and also synonyms like a thesaurus.Lemmatiation using the wordnet corpus can reduce words to words that do not have the same character structure e.g. it can reduce _better_ $\rightarrow$ _good_ if we tell it that _better_ here is a __verb__.<jupyter_code>lemmatizer = WordNetLemmatizer()
print(lemmatizer.lemmatize('better', pos='a'))
# reducing a word to a form with dissimilar charaters isn't possible in Stemming
stemmer = PorterStemmer()
print(stemmer.stem_word('better'))<jupyter_output>better
{'fraction_non_alphanumeric': 0.10265993167683728, 'fraction_numerical': 0.009547724021139303, 'mean_word_length': 2.8604598737601443, 'pattern_counts': {'":': 5, '<': 58, '<?xml version=': 0, '>': 91, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '64067', 'license': 'MIT', 'n_tokens_mistral': 10481, 'n_tokens_neox': 9874, 'n_words': 3695, 'path': 'notebook/lemmatization.ipynb', 'repo_name': 'anishLearnsToCode/lemmatization', 'size': '34286'}