<jupyter_start><jupyter_text>1) Stemming Stemming works fairly well in most of the cases but unfortunately English has so many exceptions where a more sophisticated process is required**SpaCy dosen't include stemming**, it uses lemmatization instead Stemming is basically **removes the suffixes** from a word and reduce it to its root word.For example: “Flying” is a word and its suffix is “ing”, if we remove “ing” from “Flying” then we will get base word or root word which is “Fly”. we will use Natural Language Toolkit (nltk) to understand and learn stemming  Porter Stemmer<jupyter_code>import nltk
from nltk.stem.porter import PorterStemmer
p_stemmer = PorterStemmer() # object of class PorterStemmer
words = ['run','runner','running','ran','runs','easily','fairly'] # list of words 
for word in words:
  print(word + '------>' + p_stemmer.stem(word))<jupyter_output>run------>run
runner------>runner
running------>run
ran------>ran
runs------>run
easily------>easili
fairly------>fairli<jupyter_text>Snowball Stemmer<jupyter_code># Snowball Stemmer is also called the "English Stemmer" or "Porter2 Stemmer"
# It offers a slight improvement over the original Porter stemmer
from nltk.stem.snowball import SnowballStemmer
# Pass a language parameter
s_stemmer = SnowballStemmer(language='english')
words = ['run','runner','running','ran','runs','easily','fairly']
for word in words:
    print(word +' --> '+ s_stemmer.stem(word))<jupyter_output>run --> run
runner --> runner
running --> run
ran --> ran
runs --> run
easily --> easili
fairly --> fair<jupyter_text>2) Lemmatization Lemmatization is **the Process of converting words into their dictionary form**Lemmatization considers full vocabulary of a language to apply a morphological analysise.g.Word: Feet, Lemma: Foot**Stemming**It is the process of converting words into their non-changing portion<jupyter_code>import spacy
nlp = spacy.load('en_core_web_sm')
doc1 = nlp("The striped bats are hanging on their feet for best")
for token in doc1:
    print(token.text, '\t', token.pos_, '\t',token.lemma_)

# token, POS, Lemma<jupyter_output>The 	 DET 	 the
striped 	 VERB 	 stripe
bats 	 NOUN 	 bat
are 	 AUX 	 be
hanging 	 VERB 	 hang
on 	 ADP 	 on
their 	 DET 	 -PRON-
feet 	 NOUN 	 foot
for 	 ADP 	 for
best 	 ADJ 	 good<jupyter_text>3) Stemming vs Lemmaatization<jupyter_code>doc2 = ['the','striped','bats','are','hanging','on','their','feet','for','best']
for word in doc2:
    print(word +' --> '+ s_stemmer.stem(word))
# lemma,      are ---> be
# stemming,   are ---> are

# lemma,      feet ---> foot
# stemming,   feet ---> feet

# lemma,      best ---> good
# stemming,   best ---> best<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13104988830975428, 'fraction_numerical': 0.0029784065524944155, 'mean_word_length': 4.237816764132553, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '308149', 'license': 'MIT', 'n_tokens_mistral': 931, 'n_tokens_neox': 855, 'n_words': 289, 'path': '03StemmingAndLemmatization.ipynb', 'repo_name': 'MBadriNarayanan/NaturalLanguageProcessing', 'size': '2686'}