<jupyter_start><jupyter_text>Ex 1. Write a python script that reads the data_in.csv from every cell in column labeled as comment and perform sentence tokenization and redirects in to column of data_out.csv. Perform the NE Chunking on these sentences.<jupyter_code>import pandas as pd
import nltk
nltk.download('punkt')
nltk.download('averaged_perceptron_tagger')
nltk.download('maxent_ne_chunker')
from nltk.chunk import ne_chunk
from nltk.tag import pos_tag
from nltk.tokenize import sent_tokenize
from nltk.tokenize import word_tokenize




data_in=pd.read_csv('data_in.csv')
data_in
for i in data_in['Comment']:
    sentence = sent_tokenize(i)
    data_sentence =pd.DataFrame(sentence)
    data_sentence.to_csv('data_out.csv', mode='a', header=False,index=False)
data_out=pd.read_csv('data_out.csv',header=None)
data_out
data_out.rename({0:'comment'},axis=1,inplace=True)
data_out
for i in data_out['comment']:
    chunk_sentence = ne_chunk(pos_tag(word_tokenize(i)))
    print(i)
    print(chunk_sentence)
    print("_"*50)<jupyter_output>"Hello there, how are you?
(S ``/`` Hello/NNP there/RB ,/, how/WRB are/VBP you/PRP ?/.)
__________________________________________________
Weather is awesome.
(S (GPE Weather/NNP) is/VBZ awesome/JJ ./.)
__________________________________________________
Its raining here now."
(S Its/PRP$ raining/VBG here/RB now/RB ./. ''/'')
__________________________________________________
"Hello Mr. Raja, how are you?
(S
  ``/``
  (PERSON Hello/NNP Mr./NNP Raja/NNP)
  ,/,
  how/WRB
  are/VBP
  you/PRP
  ?/.)
__________________________________________________
Weather is awesome.
(S (GPE Weather/NNP) is/VBZ awesome/JJ ./.)
__________________________________________________
Its raining here now."
(S Its/PRP$ raining/VBG here/RB now/RB ./. ''/'')
__________________________________________________
"Hello Mr. Raja, how are you.
(S
  ``/``
  (PERSON Hello/NNP Mr./NNP Raja/NNP)
  ,/,
  how/WRB
  are/VBP
  you/PRP
  ./.)
__________________________________________________
Weather is bad.
(S (GPE Weather/NNP) [...]<jupyter_text>Ex 2. Write a python script that reads the data_in.csv from every cell in column labeled as comment and perform word tokenization and redirects in to column of data_out.csv<jupyter_code>for i in data_in['Comment']:
    word = word_tokenize(i)
    sent_word =pd.DataFrame(word)
    sent_word.to_csv('data_out2.csv', mode='a', header=False,index=False)
data_out2=pd.read_csv('data_out2.csv',header=None)
data_out2
data_out2.shape<jupyter_output><empty_output><jupyter_text>Ex3. From an input file data.txt it is required to identify the POS-Tagging and display it on tree structure.<jupyter_code>data_in=open('NLPdataEx3&4-data_in.txt',"r")
for line in data_in:
    print(line.strip())
from IPython.display import Image, display
from nltk.draw import TreeWidget
from nltk.draw.util import CanvasFrame
with open('NLPdataEx3&4-data_in.txt',"r") as data_in:
    for sent in data_in:
        chunk_sentence = ne_chunk(pos_tag(word_tokenize(sent)))
        print(sent)
        print(chunk_sentence)
        chunk_sentence.draw()
        print("_"*50)<jupyter_output>Hello there, how are you? Weather is awesome. Its raining here now.

(S
  (GPE Hello/NNP)
  there/RB
  ,/,
  how/WRB
  are/VBP
  you/PRP
  ?/.
  Weather/''
  is/VBZ
  awesome/JJ
  ./.
  Its/PRP$
  raining/VBG
  here/RB
  now/RB
  ./.)
__________________________________________________
Hello Mr. Raja, how are you? Weather is awesome. Its raining here now.

(S
  (PERSON Hello/NNP)
  (PERSON Mr./NNP Raja/NNP)
  ,/,
  how/WRB
  are/VBP
  you/PRP
  ?/.
  Weather/''
  is/VBZ
  awesome/JJ
  ./.
  Its/PRP$
  raining/VBG
  here/RB
  now/RB
  ./.)
__________________________________________________
Hello Mr. Raja, how are you. Weather is bad. Its heavily raining here now.

(S
  (PERSON Hello/NNP)
  (PERSON Mr./NNP Raja/NNP)
  ,/,
  how/WRB
  are/VBP
  you/PRP
  ./.
  Weather/CC
  is/VBZ
  bad/JJ
  ./.
  Its/PRP$
  heavily/RB
  raining/VBG
  here/RB
  now/RB
  ./.)
__________________________________________________
NLP is great technique. It is nice to learn this technique.

(S
  (ORGANIZATION NLP[...]<jupyter_text>Ex 4. For a given text file exclude the stop words and perform the Stemming & lemmatization and compare the results.<jupyter_code>from nltk.tokenize import word_tokenize
from nltk.tag import pos_tag
from nltk.corpus import stopwords
from nltk import stem
from nltk.stem import PorterStemmer

ps = PorterStemmer()
wn_lemat = stem.WordNetLemmatizer()

stop_words = stopwords.words('english')

with open('data_in.txt',"r") as f:
    for sent in f:
        word_tokens=word_tokenize(sent)
        token=[]
        for word in word_tokens:
            token=token+[word]
       
        x = [word for word in token if word not in stop_words]
        
        words=[]
        words = [word for word in x if word.isalpha()]
        
        print(sent)
        print("{:10} {:10} {:10} ".format('Acutal Word |','Stemming |','Lemmatization'))
        
        for word in words:
            print("{:14} {:14} {:13} ".format(word, ps.stem(word), wn_lemat.lemmatize(word)))            
        print("--------------------------------------------\n")<jupyter_output>Comment

Acutal Word | Stemming | Lemmatization 
Comment        comment        Comment       
--------------------------------------------

"""Hello there, how are you? Weather is awesome. Its raining here now."""

Acutal Word | Stemming | Lemmatization 
Hello          hello          Hello         
Weather        weather        Weather       
awesome        awesom         awesome       
Its            it             Its           
raining        rain           raining       
--------------------------------------------

"""Hello Mr. Raja, how are you? Weather is awesome. Its raining here now."""

Acutal Word | Stemming | Lemmatization 
Hello          hello          Hello         
Raja           raja           Raja          
Weather        weather        Weather       
awesome        awesom         awesome       
Its            it             Its           
raining        rain           raining       
--------------------------------------------

"""Hello Mr. Raja, how are you. Weather [...]<jupyter_text>Ex 5. Create a small dictionary file with required set of words with weightage attached to it with positive and negative numbers. Create a python script that analyzes the given text file and classify it as negative or positive sentiment.<jupyter_code>from nltk.tokenize import word_tokenize
sent_dict = {"hate":-1,"hatred":-2,"annoyed":-2,"annoy":-1,"annoyingly":-2,"nasty":-2,"nice":1,"excellent":2,"good":1,"wonderful":2,"best":3,"better":2,"awesome":2,"beautiful":2,"beauty":1,"beautifully":2,"supreme":3}
with open('NLPdataEx5data_senti_analyze.txt','r') as sent_analyze:
    for sent in sent_analyze:
        words=word_tokenize(sent)
        score=0
        for word in words:
            if(word.lower() in sent_dict.keys()):
                score=score+sent_dict[word]  

        sentiment=""
        if(score>0):
            sentiment='Positive'
        else:
            sentiment='Negative'
        print("{}\n ---->Score is {} and Sentiment is {}\n".format(sent,score,sentiment))<jupyter_output>Rose is beautiful.

 ---->Score is 2 and Sentiment is Positive

Place is nasty to stay.

 ---->Score is -2 and Sentiment is Negative

This is the beauty of this technique.

 ---->Score is 1 and Sentiment is Positive

Concept is explained beautifully in this book.

 ---->Score is 2 and Sentiment is Positive

He annoyed me.

 ---->Score is -2 and Sentiment is Negative

Its the supreme place to stay.

 ---->Score is 3 and Sentiment is Positive

I hate this place.

 ---->Score is -1 and Sentiment is Negative

Dont annoy the customer.

 ---->Score is -1 and Sentiment is Negative

He has given nasty comments about his stay.

 ---->Score is -2 and Sentiment is Negative

Dessert is awesome.

 ---->Score is 2 and Sentiment is Positive

Your gift is wonderful.
 ---->Score is 2 and Sentiment is Positive
{'fraction_non_alphanumeric': 0.12845751308248193, 'fraction_numerical': 0.007849489160229255, 'mean_word_length': 2.788107597923549, 'pattern_counts': {'":': 17, '<': 17, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '475143', 'license': 'MIT', 'n_tokens_mistral': 2672, 'n_tokens_neox': 2549, 'n_words': 738, 'path': 'AI ML/AI & ML Basic Concepts/2-Natural_Language_Processing.ipynb', 'repo_name': 'vaibhavkrishna-bhosle/Trendnxt-Projects', 'size': '8026'}