<jupyter_start><jupyter_text>Part 1  Imports<jupyter_code>import re   #Regular Expression
import matplotlib.pyplot as plt
import pandas as pd   #Pandas to Handle Data
import numpy as np     
import nltk    #Natural Language ToolKit for Tokenizing  Word
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords # Helps to remove common words such as pronouns and adjective to clean out data
nltk.download('stopwords')
from tqdm import tqdm #Importing tqdm to track progress<jupyter_output>[nltk_data] Downloading package stopwords to C:\Users\Maaz
[nltk_data]     Karim\AppData\Roaming\nltk_data...
[nltk_data]   Package stopwords is already up-to-date!<jupyter_text>Handling the dataset Here is How I handle Data for this dataset<jupyter_code>data=pd.read_csv("C:/Users/<NAME>/datasets/datasets/datasetV3.csv")
data.head()<jupyter_output><empty_output><jupyter_text>Here we can see that we have a number of trending topics with 10 tweets. Now let us pre-process the data by removing StopWords.<jupyter_code>swe_nltk=stopwords.words('english') #Saving all stopwords into a python list
df=pd.DataFrame(data)   #Converting data into a DataFrame
for i in range(df.shape[0]):
    for j in range(9):
        text=str(df.loc[i,str('tweet'+str(j))])
        words = [word for word in text.split() if word.lower() not in swe_nltk]
        new_text = " ".join(words)
        
        df.loc[i,str('tweet'+str(j))]=new_text
df.head()<jupyter_output><empty_output><jupyter_text>Now We can see stopwords has been removed.  Building functions<jupyter_code>#Installing PyTorch
!pip install torch==1.10.1+cu113 torchvision==0.11.2+cu113 torchaudio===0.10.1+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
    
#Installing Transformers
!pip install transformers
from transformers import AutoTokenizer, AutoModelForSequenceClassification 
import torch
# Tokenizing tokens from a pre-trained model

tokenizer = AutoTokenizer.from_pretrained('nlptown/bert-base-multilingual-uncased-sentiment')

#Using Pre-trained model for Classification
model = AutoModelForSequenceClassification.from_pretrained('nlptown/bert-base-multilingual-uncased-sentiment')
# Making a sentiment_score function which will return us the value between 0 and 5. 
def sentiment_score(review):
    tokens = tokenizer.encode(review, return_tensors='pt')
    result = model(tokens)
    return int(torch.argmax(result.logits))+1
df['score']=0
for i in tqdm(range(1000)):
    for j in range(9):
        df.loc[i,'score']=sentiment_score(df.loc[i,str('tweet'+str(j))])+df.loc[i,'score']
df['score']=df['score']/10 #Calculating average score
df.head()<jupyter_output><empty_output><jupyter_text>Now we can see The scores of the tweets of each trending topic the lesser the score the more disliked topic it is.<jupyter_code># To find most liked topic of the day.
print("Liked Topic of the day is " + str(df.loc[df['score'].idxmax(),'Data']) +" with the score of "+ str(df.loc[df['score'].idxmax(),'score']) )
# To find most disliked topic of the day.
print("Most disliked Topic of the day is " + str(df.loc[df['score'][0:999].idxmin(),'Data']) +" with the score of "+ str(df.loc[df['score'][0:999].idxmin(),'score']))<jupyter_output>Liked Topic of the day is  HBDSarathkumar with the score of 4.5
Most disliked Topic of the day is  VizagBlast with the score of 0.9<jupyter_text>We can conclude that in the top 1000 tweets the most Liked topic is HDBSarathKumar with avg score of 4.5 and most disliked topic in top 1000 tweets is VizagBlast with avg score of 0.9.  Part 2  Plots  For Top 10 Tweets<jupyter_code>#code
plt.plot(list(range(1,11)),df['score'][0:10])
plt.xlabel('Rank of Trending Tweets')
plt.ylabel('Score out of 5')
plt.title('Score vs Top 10 Trending Tweets')<jupyter_output><empty_output><jupyter_text>Now let us see score using bar charts.<jupyter_code>x_label=(list(range(0,6)))
y_label=[round(num) for num in df['score'][0:10]]
import collections
y_label=collections.Counter(y_label)
li = [0,0,0,0,0,0]
index=0
for i in y_label.keys():
    li[i]=y_label[i]
plt.bar(x_label,li)
import statistics
statistics.mean(df['score'][0:10])<jupyter_output><empty_output><jupyter_text>Top 1000 Tweets Now let us see the graph for first 1000 tweets<jupyter_code>plt.plot(list(range(1,1001)),df['score'][0:1000])
plt.xlabel('Rank of Trending Tweets')
plt.ylabel('Score out of 5')
plt.title('Score vs Top 1000 Trending Tweets')<jupyter_output><empty_output><jupyter_text>Now let us check bar chart for first 1000 tweets<jupyter_code>x_label=(list(range(0,6)))
y_label=[round(num) for num in df['score'][0:1000]]
y_label=collections.Counter(y_label)
li = [0,0,0,0,0,0]
for i in list(y_label.keys()):
    li[i]=y_label[i]
plt.bar(x_label,li)
statistics.mean(df['score'][0:1000])<jupyter_output><empty_output><jupyter_text>Let us see the variation of scores as our value of tweets increases<jupyter_code>x=[1,10,50,100,200,300,500,700,900,1000]
y=[]
for i in x:
    y.append(statistics.mean(df['score'][0:i]))
plt.plot(x,y)<jupyter_output><empty_output><jupyter_text>Conclusion<jupyter_code>#conclude with your observations<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12531670239719353, 'fraction_numerical': 0.03215747417657377, 'mean_word_length': 6.127777777777778, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '243505', 'license': 'MIT', 'n_tokens_mistral': 1956, 'n_tokens_neox': 1825, 'n_words': 471, 'path': 'CONTRIBUTION/Jupyter notebooks/Maaz_Karim.ipynb', 'repo_name': 'Rudransh1084/public-sentiment-analysis-based-on-twitter-hashtags', 'size': '5135'}