<jupyter_start><jupyter_text>Lemmatization<jupyter_code>import nltk
nltk.download()
paragraph = """In a country like India with a galloping population, unfortunately nobody is paying attention to the issue 
of population. Political parties are feeling shy, politicians are feeling shy, Parliament also does not adequately discuss 
about the issue,” said Naidu while addressing the 58th convocation of Indian Agricultural Research Institute (IARI).

He said, “You know how population is growing, creating problems. See the problems in Delhi, traffic, more human beings, 
more vehicles, more tension, less attention. If you have tension you cannot pay attention.” 
Emphasising on the need to increase food production to meet demand of growing population, Naidu said, 
“In future if population increases like this, and you are not able to adequately match it with increase in production, 
there will be problem"""
sentences = nltk.sent_tokenize(paragraph)
# sentences
from nltk.stem import WordNetLemmatizer
from nltk.corpus import stopwords
lemmatization = WordNetLemmatizer()
for i in range(len(sentences)):
    words = nltk.word_tokenize(sentences[i])
    words = [lemmatization.lemmatize(word) for word in words if word not in set(stopwords.words('english'))]
    sentences[i] = ' '.join(words)
sentences<jupyter_output><empty_output><jupyter_text>Difference between Stemming and Lemmatization  - Lemmatization words have meaning whereas Stemming words may or may not have meaning. - Lemmatization process is slow compared to Stemming. - Lemmatization is preffered when meaning of word is crucial/important for example in filling applications    whereas Stemming is preffered when the meaning of the word is least important like email is spam or not<jupyter_code># Lemmatization
import nltk
nltk.download()
# Paragraph or Corpse
paragraph = """In a country like India with a galloping population, unfortunately nobody is paying attention to the issue 
of population. Political parties are feeling shy, politicians are feeling shy, Parliament also does not adequately discuss 
about the issue,” said Naidu while addressing the 58th convocation of Indian Agricultural Research Institute (IARI).

He said, “You know how population is growing, creating problems. See the problems in Delhi, traffic, more human beings, 
more vehicles, more tension, less attention. If you have tension you cannot pay attention.” 
Emphasising on the need to increase food production to meet demand of growing population, Naidu said, 
“In future if population increases like this, and you are not able to adequately match it with increase in production, 
there will be problem"""
# Tokenization
sentences = nltk.sent_tokenize(paragraph)
# sentences
# Import Libraries
from nltk.stem import WordNetLemmatizer
from nltk.corpus import stopwords
# Tokenization & Lemmatization objective creation
Lemmatization = WordNetLemmatizer()
for i in range(len(sentences)):
    words = nltk.word_tokenize(sentences[i])
    words = [Lemmatization.lemmatize(word) for word in words if word not in set(stopwords.words('english'))]
    sentences[i] = ' '.join(words)
sentences<jupyter_output><empty_output><jupyter_text>Difference between Stemming & Lemmatization<jupyter_code># 1. Lemmatization words have meaning whereas Stemming words may or may not have meaning.
# 2. Lemmatization process is slow compared to Stemming.
# 3. Lemmatization is preffered when meaning of words is crucial/important fr example in filling applications 
# whereas Stemming is preferred when meaning of word is least important like email is spam or not.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05687465488680287, 'fraction_numerical': 0.001932633903920486, 'mean_word_length': 5.527927927927928, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '76653', 'license': 'Unlicense', 'n_tokens_mistral': 971, 'n_tokens_neox': 947, 'n_words': 447, 'path': '003-vk_NLP - Lemmatization.ipynb', 'repo_name': 'vitthalkcontact/NLP', 'size': '3622'}