<jupyter_start><jupyter_text>Run in Google Colab<jupyter_code>import numpy as np
import random
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.metrics import f1_score

import json, urllib.request<jupyter_output><empty_output><jupyter_text>Load Data  Data Class Our first model will be automatically classifying positive and negative comments Creating data to train the models is not an good approach, getting data by some other sources or by web crawling is one the best techniques, for negative and positive sentence data you can craw the amazon's review column of any product.<jupyter_code># Storing the Path of file in a variable
project_url = 'https://raw.githubusercontent.com/gimseng/99-ML-Learning-Projects/master/005/'
file_name = project_url+'data/books_small_10000.json'

# Opening JSON file and reading it line by line.
with urllib.request.urlopen(file_name) as f:
    for line in f:
        review = json.loads(line)
        # Getting review text
        print(review['reviewText'])
        # Getting the Overall rating
        print(review['overall'])
        break
# Storing the Path of file in a variable
file_name = project_url+'data/books_small_10000.json'

# Create empty list to store tuple objects of every data
reviews = []

# Opening JSON file and reading it line by line.
with urllib.request.urlopen(file_name) as f:
    for line in f:
        review = json.loads(line)
        reviews.append((review['reviewText'], review['overall']))
# Printing Random object from the reviews
reviews[5]
reviews[5][1]
# Now creating a Data class for our reviews and we are gonn initialise it with text and score
# Now insted of appending this tuple we will create a Review object and pass in text and score.

class Review:
    def __init__(self, text, score):
        self.text = text
        self.score = score
        

        # Storing the Path of file in a variable
file_name = project_url+'data/books_small_10000.json'

# Create empty list to store tuple objects of every data
reviews = []

# Opening JSON file and reading it line by line.
with urllib.request.urlopen(file_name) as f:
    for line in f:
        review = json.loads(line)
        reviews.append(Review(review['reviewText'], review['overall']))

# Getting score 
reviews[5].score
reviews[5].text
import random

class Sentiment:
    NEGATIVE = 'NEGATIVE'
    NEUTRAL = 'NEUTRAL'
    POSITIVE = 'POSITIVE'

class Review:
    def __init__(self, text, score):
        self.text = text
        self.score = score
        # initialising sentiments 4/5 stars means +ve and 1/2 stars means -ve
        self.sentiments = self.get_sentiments()
        
    def get_sentiments(self):
        if self.score <= 2:
            return Sentiment.NEGATIVE
        elif self.score >= 4:
            return Sentiment.POSITIVE
        elif self.score == 3:
            return Sentiment.NEUTRAL
        

class ReviewContainer:
    def __init__(self, reviews):
        self.reviews = reviews
        
    def get_text(self):
        [x.text for x in training]
        
    def get_sentiment(self):
        [x.sentiment for x in self.sentiment]
    
    def evenly_distribute(self):
        negative = list(filter(lambda x: x.sentiments == Sentiment.NEGATIVE, self.reviews))
# Its looking all the reviews mapping every sentiment, its basically filtering based upon negative sentiments, keeping 
# track of that in the negative list
        positive = list(filter(lambda x: x.sentiments == Sentiment.POSITIVE, self.reviews))
# Distribute evenly in the prep Data cell 10.
        positive_shrunk = positive[:len(negative)]
        self.reviews = negative + positive_shrunk
        # Shuffle so u wont know what comes when
        random.shuffle(self.reviews)<jupyter_output><empty_output><jupyter_text>Load Data<jupyter_code># Storing the Path of file in a variable
file_name =project_url+ 'data/books_small_10000.json'

# Create empty list to store tuple objects of every data
reviews = []

# Opening JSON file and reading it line by line.
with urllib.request.urlopen(file_name) as f:
    for line in f:
        review = json.loads(line)
        reviews.append(Review(review['reviewText'], review['overall']))
reviews[5].sentiments<jupyter_output><empty_output><jupyter_text>Now the Machine learning algorithms loves the numerical data and its kinda hard to work with the strings,So what we are gonna doing here, we will be using count vectoriser here and break down the sentence in a dictionarylike we have two sentences,1. This book is great !2. This book was so bad.So the dictionary of the words will include This, book, is, great, was, so, bad.so we will map these dict with the sentences itself to see what words does a sentence have so                            This book is great was so bad    1. This book is great !  1 |  1 | 1 |   1 | 0 |0 | 0|    2. This book was so bad  1 |  1 | 0 |   0 | 1 |1 | 1|    3. Was a great book      0 |  1 | 0 |   1 | 1 |0 | 0|so 1 means sentence have that word and 0 means sentence doesnt have that word, 3 rd sentence is that sentence we have never seen before but we can also map that using the knowlegede of previous words in the dictionary but we cant handle 'a' here because that is not included in the dictionary during the training time.<jupyter_code># Using train test split to split data into test and training
# What list you are passing here you will get 2 times of that

training, test = train_test_split(reviews, test_size = 0.33, random_state = 42)

cont = ReviewContainer(training)
# We will use evenly distribute method.

cont.evenly_distribute()

len(training)
# Now we build a classifier on that training set and after that we will test everything on our test data
# We will have to pass our tarining data into the vectorizer, as we have to take text and predict if it is +ve or -ve 
# So what we are gonna pass into vectorizer is X which is our sentence and y = sentiments corresponding to that.
train_x = [x.text for x in training]
train_y = [x.sentiments for x in training]


test_x = [x.text for x in test]
test_y = [x.sentiments for x in test]

# We are getting the same text again
# train_x[0]
# train_y[0]<jupyter_output><empty_output><jupyter_text>Bag of Words Vectorization In order to perform machine learning on text documents, we first need to turn the content into numerical feature vectors<jupyter_code>vectorizer = CountVectorizer()

# Transforming String Data to numerical data.
# Now this is the main data we want to use while training.
train_x_vectors = vectorizer.fit_transform(train_x) # These are 2 steps fit and transform, we can also do them individually.
test_x_vectors = vectorizer.transform(test_x) # We just wanna transform the test data not to fit that.


# So now our main data will be train_x_vectors and train_y and we wanna fit our data around these.<jupyter_output><empty_output><jupyter_text>Classification  Linear SVM<jupyter_code>from sklearn import svm

clf_svm = svm.SVC(kernel = 'linear')
# Fitting the data.
clf_svm.fit(train_x_vectors, train_y)

# Prediction using testing data

clf_svm.predict(test_x_vectors[0])


# If we read this test data which we have predicted we will truely identify it as positive, so our data is fitted pretty well
print(test_x[0])<jupyter_output>was sent an Arc of this book for an honest review and here it is = This is the kind of book that you want to read while sitting in front of the fire with a cup of hot apple cider and a blanket over your legs.I have read many of <NAME>'s books and have never been disappointed. This first book in her new Hope series does not disappoint either.This is the story of Emma, a new vet who has come back home to open her own practice and <NAME>, a police officer in the same town.Both have been previously burned by love so both have issues but, that doesn't stop them from acting on that attraction.This book pulls you in from the first page, wraps you up and doesn't let you go until the end.I loved it!<jupyter_text>Decision Tree<jupyter_code>from sklearn.tree import DecisionTreeClassifier

clf_dec = DecisionTreeClassifier()

# fitting the same exact data we did earlier
clf_dec.fit(train_x_vectors, train_y)
clf_dec.predict(test_x_vectors[0])
# It is working perfect<jupyter_output><empty_output><jupyter_text>Naive Bayes<jupyter_code>from sklearn.naive_bayes import GaussianNB

clf_gnb = GaussianNB()

trainingset = np.array(train_x_vectors)

clf_gnb.fit(train_x_vectors.todense(), train_y)

# clf_gnb.predict(test_x_vectors[0])<jupyter_output><empty_output><jupyter_text>Logistic Regression<jupyter_code>from sklearn.linear_model import LogisticRegression

clf_lr = LogisticRegression(max_iter = 500)

clf_lr.fit(train_x_vectors, train_y)
clf_lr.predict(test_x_vectors[0])
# This time it is also fitted well<jupyter_output><empty_output><jupyter_text>Evaluation<jupyter_code>clf_svm.score(test_x_vectors, test_y)
clf_lr.score(test_x_vectors, test_y)
clf_dec.score(test_x_vectors, test_y)
clf_gnb.score(test_x_vectors.toarray(), test_y)
# toarray() is used to pack the sparse matrix into dense one.<jupyter_output><empty_output><jupyter_text>These all were the mean accuracies of our models but now we will have a look upon the f1 score<jupyter_code># F1_scores
# Each individual class will gets its own f1 score
# so what does these means , lets pass in the labels
print(f1_score(test_y, clf_svm.predict(test_x_vectors), average = None, 
      labels = [Sentiment.POSITIVE, Sentiment.NEGATIVE ,Sentiment.NEUTRAL]))
print(f1_score(test_y, clf_lr.predict(test_x_vectors), average = None, 
      labels = [Sentiment.POSITIVE, Sentiment.NEGATIVE ,Sentiment.NEUTRAL]))
print(f1_score(test_y, clf_dec.predict(test_x_vectors), average = None, 
      labels = [Sentiment.POSITIVE, Sentiment.NEGATIVE ,Sentiment.NEUTRAL]))<jupyter_output>[0.90738061 0.40268456 0.2656    ]
[0.92315808 0.42016807 0.29496403]
[0.87511135 0.16368286 0.16778523]<jupyter_text>So the svm is predicting POSITIVE 90% and negative for 40% time correctly and similarly we can see others performing, so we can conclude that best classifier would be the logistic regression here, now we will work on how we wanna improve f1 scores for NEGATIVE Sentiment.<jupyter_code>print(train_y.count(Sentiment.POSITIVE)) # No. of positive responses in the data

print(train_y.count(Sentiment.NEGATIVE)) # No. of negative responses in the data

# So due to having many positive responses our model is baised around the positives.<jupyter_output>5611
436
{'fraction_non_alphanumeric': 0.07053351053730776, 'fraction_numerical': 0.017277387507119803, 'mean_word_length': 4.174361493123772, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '425003', 'license': 'MIT', 'n_tokens_mistral': 3348, 'n_tokens_neox': 3158, 'n_words': 1250, 'path': '005/solution/sentiment_analysis.ipynb', 'repo_name': 'Satyajit99p/99-ML-Learning-Projects', 'size': '10547'}