<jupyter_start><jupyter_text>Data Science ApplicationsAssignment 4Part : A Instruction:This assignment should be completed individually using either Python or R. Upon completion, present your result in one submission, including the report, answers generated and plots. Also, submit the source codes used to generate your results as a separate attachment.Part A:  Topic Modeling & Sentiment Analysis (50 Points)Sentiment analysis is useful for identifying attitudes in text data about a subject of interest. It is scored using polarity values that range from 1 (i.e., more positivity) to -1 (i.e., more negativity). In this part, you will:A. Estimate the public's perception of natural hazards using tweets from the tornado dataset providedUsing the textblob package in python, calculate a polarity value for each tweet (10 points) Plot the polarity values in a histogram to identify the overall sentiment (10 points) What does the histogram of polarity values tell you about sentiments in the tweets? Are they positive or negative? (5 points)B.  Develop a text summarizer Preprocess the tweets & compute frequency distribution for most common 10 words (10 points)  Create a topic model using the LDA approach & extract 10 meaningful topics (15 points) **Importing required Libraries.**<jupyter_code># Install Libraries
!pip install textblob
!pip install tweepy
!pip install contractions
#Import all the important required libraries 
import pandas as pd
import numpy as np
import spacy
import matplotlib.pyplot as plt
import warnings
import os
import re
import contractions
import nltk
import gensim
import seaborn as sns

nltk.download('punkt')
nltk.download('wordnet')
nltk.download('stopwords')
nltk.download('vader_lexicon')

from spacy import displacy
from textblob import TextBlob
from nltk.sentiment.vader import SentimentIntensityAnalyzer
from nltk.stem import PorterStemmer, WordNetLemmatizer
from spacy.lang.en import English
from collections import Counter 
from gensim import corpora
from gensim.models import CoherenceModel
from nltk.corpus import stopwords

warnings.filterwarnings('ignore')
%matplotlib inline

#Importing data 
df = pd.read_excel("Tornado.xlsx", header=None)
data_size = len(df)
print(data_size)
print(df.head())
print(df.tail())
#Display imported data 
df
#Creating new column with combined data 
df['Text'] = df[df.columns[0:]].apply(
    lambda x: ','.join(x.dropna().astype(str)),
    axis=1
)
#Displaying Combined data 
df<jupyter_output><empty_output><jupyter_text>**1. Using the textblob package in python, calculate a polarity value for each tweet (10 points)**<jupyter_code>sentiment_ob = [TextBlob(text) for text in df['Text']]
polarity = [sen.sentiment.polarity for sen in sentiment_ob]
print(polarity)<jupyter_output>[-0.2916666666666667, -0.36458333333333337, -0.1, 0.6, 0.44999999999999996, 0.16666666666666666, -0.5, -0.4, 0.2, 0.0, 0.0, 0.6, -0.15833333333333333, 0.625, -0.1375, 0.2125, 0.14166666666666666, -0.1, 0.25, -0.152587890625, -0.125, -0.125, 0.2, 0.625, 0.125, 0.4, 0.2, 0.3666666666666667, 0.6000000000000001, 0.2, -0.25, 0.38125, -0.125, -0.07500000000000001, -0.07500000000000001, 0.08333333333333333, -0.1, 0.2, 0.05000000000000002, 0.5, -0.5833333333333334, -0.05, 0.75, -0.30185185185185187, 0.8, -0.625, 1.0, 0.5, 0.25, 1.0, -0.25, 0.2869318181818182, 0.4, -0.25, -0.4, 0.1787878787878788, -0.1125, 0.5, 0.34285714285714286, 0.5083333333333333, -0.4, -0.5, -0.025000000000000022, 0.4, 0.0, 0.04999999999999999, 0.6825, 0.2, 0.375, -0.2, 0.5, 0.56875, 0.56875, 0.2869318181818182, 1.0, 1.0, 0.16818181818181818, 0.13636363636363635, 0.7, 0.13636363636363635, 0.4, 0.1, 0.13636363636363635, 0.13636363636363635, 0.25757575757575757, 0.1, 0.5, -0.1, -0.026767676767676756, 0.23333333333333336, 0.5[...]<jupyter_text>**2. Plot the polarity values in a histogram to identify the overall sentiment (10 points)**<jupyter_code>#Plotting graph for Polarity without preprocessing the data
plt.hist(polarity, bins=15)
plt.show()<jupyter_output><empty_output><jupyter_text>**3. What does the histogram of polarity values tell you about sentiments in the tweets? Are they positive or negative? (5 points)** **Ans :** The polarity histogram is plotted from -1 to 1, with a plot from -1 to 0 representing negative sentiment tweets and a plot from 0 to +1 representing positive sentiment tweets. Tweets with a sentiment of 0 are considered neutral. If we carefully observe the histogram map, we can see that the mean of the bell curve varies from 0 to 0.15 polarity magnitude, implying that the majority of tweets have a positive feeling.The histogram of polarity values is tilted to the left and fits the Gaussian distribution/Normal distribution, suggesting that the bulk of total tweets are optimistic in nature. The second metric compares the cumulative number of tweets for positive and negative polarity/sentiment. The overall number of positive sentiments outnumbers the total number of negative sentiments. **B. Develop a text summarizer** **Preprocessing the data**<jupyter_code>df['Text'] = [re.sub(r'@\w+|http\S+|www\S+|#\S+',"",line.lower()) for line in df['Text']]
punc = set([word for line in df['Text'] for word in line if not word.isalnum()])
punc.remove(' ')
punc.remove("'")
remove = '|'.join('\\'+i for i in punc)
df['Text'] = df['Text'].apply(lambda x: re.sub(remove, ' ', x))
def remove_numbers_punctuation(text):
    return re.sub('[^a-zA-Z,,.!:;]' , ' ', text)

def remove_brackets(text):
    return re.sub('\[[^]]*\]', "", text)

def remove_num_brac_punct(text):
    text = remove_numbers_punctuation(text)
    text = remove_brackets(text)
    return text

df['Text'] = df['Text'].apply(remove_num_brac_punct)
df['Text']
for text in df['Text'].head():
    print(text)
sentiment_ob_after_clean = [TextBlob(text) for text in df['Text']]
polarity_after_clean = [sen.sentiment.polarity for sen in sentiment_ob_after_clean]<jupyter_output><empty_output><jupyter_text>**Plotting graph of polarity after preprocessing the data**<jupyter_code>plt.hist(polarity_after_clean, bins=15)
plt.show()
d = contractions.contractions_dict
df['Text'] = df['Text'].apply(lambda x: ' '.join([d[wor] if wor in d.keys() else wor for wor in x.split()]))
for text in df['Text'].head():
    print(text)
st_words = set(stopwords.words('english'))
df['Text'] = df['Text'].apply(lambda x: [word for word in x.split() if word not in st_words])
for text in df['Text'].head():
    print(text)
wnl = WordNetLemmatizer()
df['Text'] = df['Text'].apply(lambda x: [wnl.lemmatize(word) for word in x])
for text in df['Text'].head():
    print(text)<jupyter_output>['rt', 'please', 'rt', 'food', 'needed', 'shelter', 'nc', 'area', 'hit', 'hard', 'tornado', 'please', 'contact', 'u']
['please', 'rt', 'food', 'needed', 'shelter', 'nc', 'area', 'hit', 'hard', 'tornado', 'please', 'contact', 'u', 'info', 'org', 'help']
['raining', 'tornado', 'warning', 'coma', 'eating', 'food']
['lmao', 'rt', 'get', 'food', 'hope', 'tornado', 'blow', 'yall', 'food', 'yall', 'mouth']
['saved', 'tornado', 'warning', 'ready', 'eat', 'good', 'food']<jupyter_text>**1. Preprocess the tweets & compute frequency distribution for most common 10 words (10 points)**<jupyter_code>#convert lemmatized tokens into sentences 
df_lemmatize_data = ','.join([item for sub_list in df['Text'] for item in sub_list])
nlp = English()
#to increase length 
nlp.max_length = 40000000
df_lemmatize_inter = nlp(df_lemmatize_data)
# get all tokens that aren't punctuation
df_lemmatize_inter1 = [token.text for token in df_lemmatize_inter if token.is_punct != True]
# get the frequency of each word (token) in the  string
all_word_freq = Counter(df_lemmatize_inter1)
all_word_freq
# get the 10 most frequent words
top_ten_words = all_word_freq.most_common(10)
# view the 10 most common words
top_ten_words
#Converting word frequency into dataframes
frequency_df = pd.DataFrame.from_dict(all_word_freq, orient='index').reset_index()
# rename the columns to "word" and "freq"
frequency_df.columns=["word", "freq"]
#Plotting Graph for top 20 words
fig, ax = plt.subplots(figsize=(12,6))
sns.barplot(data=frequency_df.sort_values(by="freq", ascending=False).head(10), 
            y="freq", 
            x="word", 
            color='#7bbcd5')
plt.ylabel("Frequency")
plt.xlabel("Word")
plt.title("Top 10 Most Frequent Words")
sns.despine();<jupyter_output><empty_output><jupyter_text>**2. Create a topic model using the LDA approach & extract 10 meaningful topics (15 points)**<jupyter_code>dictionary = corpora.Dictionary(df["Text"].tolist())
doc_term_matrix = [dictionary.doc2bow(doc) for doc in df["Text"]]
lda = gensim.models.ldamodel.LdaModel
lda_model = lda(doc_term_matrix, num_topics=10, id2word=dictionary, passes=20)
print(lda_model.print_topics())
topics = lda_model.print_topics(num_words=10)
for topic in topics:
    print(topic)
print('\nPerplexity:', lda_model.log_perplexity(doc_term_matrix))

coherence_score_lda = CoherenceModel(model=lda_model, texts=df['Text'], dictionary=dictionary, coherence='c_v')
coherence_score = coherence_score_lda.get_coherence()

print('\nCoherence Score:', coherence_score)<jupyter_output>Perplexity: -7.1002333525111325

Coherence Score: 0.3017943511450268
{'fraction_non_alphanumeric': 0.13036859845601825, 'fraction_numerical': 0.08915950853539198, 'mean_word_length': 5.660391020999276, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 18, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '88086', 'license': 'MIT', 'n_tokens_mistral': 3706, 'n_tokens_neox': 3082, 'n_words': 962, 'path': 'Assignment_4(Part_A)__DTI5125_Rahul_Raj_Sharma.ipynb', 'repo_name': 'sharmarahulraj03/NLP_Tasks', 'size': '9197'}