<jupyter_start><jupyter_text>orfrom IPython.display import ImageImage(url= "https://miro.medium.com/max/525/1*96kuTKGS6y9_gf7MRS2gHw.png") USEFUL LINKS/CITATION - https://nycdatascience.com/blog/student-works/amazon-fine-foods-visualization/http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html  Objective:To do Sentiment Analysis and determine whether the Reviews from Amazon are Negative (rating of 1 or 2), Positive (rating of 4 or 5) or Neutral.  OVERVIEWSentiment analysis is the use of natural language processing, text analysis, computational linguistics, and biometrics to systematically identify, extract, quantify, and study affective states and subjective information.Since customers express their thoughts and feelings more openly than ever before, sentiment analysis is becoming an essential tool to monitor and understand that sentiment. Automatically analyzing customer feedback, such as opinions in survey responses and social media conversations, allows brands to learn what makes customers happy or frustrated, so that they can tailor products and services to meet their customers’ needs. For example, using sentiment analysis to automatically analyze 4,000+ reviews about your product could help you discover if customers are happy about your pricing plans and customer service.Ques- How to determine if a review is positive or negative?[Ans] We could use the Score/Rating. A rating of 4 or 5 could be cosnidered a positive review. A review of 1 or 2 could be considered negative. A review of 3 is nuetral and ignored. This is an approximate and proxy way of determining the polarity (positivity/negativity) of a review.  What is Natural Language?Natural language processing (NLP) is an area of computer science and artificial intelligence concerned with the interactions between computers and human (natural) languages, in particular how to program computers to process and analyze large amounts of natural language data.Challenges in natural language processing frequently involve speech recognition, natural language understanding, and natural language generation.Natural language processing includes many different techniques for interpreting human language, ranging from statistical and machine learning methods to rules-based and algorithmic approaches. We need a broad array of approaches because the text- and voice-based data varies widely, as do the practical applications.  ARTIFICIAL NEURAL NETWORK(ANN)An artificial neural network (ANN) is the component of artificial intelligence that is meant to simulate the functioning of a human brain.Processing units make up ANNs, which in turn consist of inputs and outputs. The inputs are what the ANN learns from to produce the desired output.EXAMPLES - Commercial applications of these technologies generally focus on solving complex signal processing or pattern recognition problems. Examples of significant commercial applications since 2000 include handwriting recognition for check processing, speech-to-text transcription, oil-exploration data analysis, weather prediction and facial recognition.  TYPESFine-grained Sentiment AnalysisIf polarity precision is important to your business, you might consider expanding your polarity categories to include:1.Very positive2.Positive3.Neutral4.Negative5.Very negative  Is spaCy or NLTK better?While NLTK provides access to many algorithms to get something done, spaCy provides the best way to do it. It provides the fastest and most accurate syntactic analysis of any NLP library released to date. It also offers access to larger word vectors that are easier to customize  WHY IS IT IMPORTANT?Sentiment analysis is extremely useful in social media monitoring as it allows us to gain an overview of the wider public opinion behind certain topics. Being able to quickly see the sentiment behind everything from forum posts to news articles means being better able to strategise and plan for the future. companies want their brand being perceived positively, or at least more positively than the brands of competitors.  BASIC STEPSThe common and most basic steps are:1.Remove URLs and email addresses from every single sample — because they won’t add meaningful value.2.Remove punctuation signs — otherwise your model won’t understand that “good!” and “good” are actually meaning the same thing.3.Lowercase all text — because you want to make the input text as generic as possible and avoid that, for example, a “Good” which is at the beginning of a phrase to be understood differently than the “good” in another sample.4.Remove stop-words — stop-words refer to the most common words in a language, such as “I”, “have”, “are” and so on. I hope you get the point because there’s not an official stop-words list out there.5.Stemming/Lemmatizing:  Lemmatizing generally returns valid words (that exist) while stemming techniques return (most of the times) shorten words, that’s why lemmatizing is used more in real world implementations. This is how lemmatizers vs. stemmers work: suppose you want to find the root word of ‘caring’: ‘Caring’ -> Lemmatization -> ‘Care’. In the other hand: ‘Caring’ -> Stemming -> ‘Car’; did you get the point? You can research about both and obviously implement any if the business requires it.6.Transform dataset (text) into numeric tensors — Usually referred as vectorization.like all other neural networks, deep-learning models don’t take as input raw text: they only work with numeric tensors, that’s why this step is not negotiable. There are multiple ways to do so; for example, if you’re going to use a classic ML model (not DL) then you definitely should go with CountVectorizer, TFIDF Vectorizer or just the basic but not so good approach: Bag-Of-Words. It’s up to you. However, if you’re going to implement Deep Learning you might know that the best way is to turn your text data (that can be understood as sequences of word or sequences of characters) into low-dimensional floating-point vectors<jupyter_code>from IPython.display import Image
from IPython.core.display import HTML 
Image(url= "https://miro.medium.com/max/650/1*P9agr-brgGGbTCjA7PEBvQ.png", width=600, height=600)<jupyter_output><empty_output><jupyter_text>ABOUT DATASETContextThis dataset consists of reviews of fine foods from amazon. The data span a period of more than 10 years, including all ~500,000 reviews up to October 2012. Reviews include product and user information, ratings, and a plain text review. It also includes reviews from all other Amazon categories.ContentsReviews.csv: Pulled from the corresponding SQLite table named Reviews in database.sqlitedatabase.sqlite: Contains the table 'Reviews'Data includes:Reviews from Oct 1999 - Oct 2012568,454 reviews256,059 users74,258 products260 users with > 50 reviewsAttribute Information:IdProductId - unique identifier for the productUserId - unqiue identifier for the userProfileNameHelpfulnessNumerator - number of users who found the review helpfulHelpfulnessDenominator - number of users who indicated whether they found the review helpful or notScore - rating between 1 and 5Time - timestamp for the reviewSummary - brief summary of the reviewText - text of the reviewDataset source : https://www.kaggle.com/snap/amazon-fine-food-reviews?select=Reviews.csv  SPACYSpacy is written in cython language, (C extension of Python designed to give C like performance to the python program). Hence is a quite fast library. spaCy provides a concise API to access its methods and properties governed by trained machine (and deep) learning models.Implementation of spacy and access to different properties is initiated by creating pipelines. A pipeline is created by loading the models. There are different type of models provided in the package which contains the information about language – vocabularies, trained vectors, syntaxes and entities.These pipelines outputs a wide range of document properties such as – tokens, token’s reference index, part of speech tags, entities, vectors, sentiment, vocabulary etc. a)Tokenization: Every spaCy document is tokenized into sentences and further into tokens which can be accessed by iterating the document.b)Part of Speech Tagging: Part-of-speech tags are the properties of the word that are defined by the usage of the word in the grammatically correct sentence. These tags can be used as the text features in information filtering, statistical models, and rule based parsing.c)Entity Detection Spacy consists of a fast entity recognition model which is capable of identifying entitiy phrases from the document. Entities can be of different types, such as – person, location, organization, dates, numerals, etc. These entities can be accessed through “.ents” property.d)Dependency Parsing One of the most powerful feature of spacy is the extremely fast and accurate syntactic dependency parser which can be accessed via lightweight API. The parser can also be used for sentence boundary detection and phrase chunking. The relations can be accessed by the properties “.children” , “.root”, “.ancestor” etc.e)Noun Phrases Dependency trees can also be used to generate noun phrasesf)Word to Vectors Integration Spacy also provides inbuilt integration of dense, real valued vectors representing distributional similarity information. It uses GloVe vectors to generate vectors. GloVe is an unsupervised learning algorithm for obtaining vector representations for words.<jupyter_code>!pip install spacy
!pip install pandas --upgrade<jupyter_output>Requirement already satisfied: pandas in c:\users\work\anaconda3\lib\site-packages (1.3.2)<jupyter_text>The plotly Python library is an interactive, open-source plotting library that supports over 40 unique chart types covering a wide range of statistical, financial, geographic, scientific, and 3-dimensional use-cases.<jupyter_code>!pip install plotly    
!pip install spacy
!pip install numpy
from IPython.core.display import display, HTML
display(HTML("<style>.container { width:100% !important; }</style>"))
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import numpy as np # linear algebra
import seaborn as sns
color = sns.color_palette()
import spacy
from spacy import displacy #spaCy also comes with a built-in dependency visualizer that lets you check your model's predictions in your browse
from spacy.util import minibatch, compounding
import warnings
warnings.filterwarnings(action="ignore")
import matplotlib.pyplot as plt
%matplotlib inline
import plotly.offline as py
py.init_notebook_mode(connected=True)
import plotly.graph_objs as go #Plotly's Python graphing library makes interactive, publication-quality graphs
import plotly.tools as tls
import plotly.express as px
from nltk.stem import PorterStemmer #The Porter stemming algorithm (or 'Porter stemmer') is a process for removing the commoner morphological and inflexional endings from words in English
!pip install textblob #TextBlob is a simple library which supports complex analysis and operations on textual data.
from textblob import TextBlob
import sqlite3
import string
# This is used for fast string concatination
from io import StringIO<jupyter_output><empty_output><jupyter_text>Gensim is a free open-source Python library for representing documents as semantic vectors, as efficiently (computer-wise) and painlessly (human-wise) as possible. Gensim is designed to process raw, unstructured digital texts (“plain text”) using unsupervised machine learning algorithms.<jupyter_code>pip install gensim
#The sklearn. metrics module implements several loss, score, and utility functions to measure classification performance. Some metrics might require probability estimates
#of the positive class, confidence values, or binary decisions values.
from sklearn import metrics
from sklearn.metrics import roc_curve, auc #The Area Under the Curve (AUC) is the measure of the ability of a classifier to distinguish between classes and is used as a summary of the ROC curve. The higher the AUC, the better the performance 
#of the model at distinguishing between the positive and negative classes.
from nltk.stem.porter import PorterStemmer

import re
# Tutorial about Python regular expressions: https://pymotw.com/2/re/
import string
from nltk.corpus import stopwords
from nltk.stem.wordnet import WordNetLemmatizer

from gensim.models import Word2Vec
from gensim.models import KeyedVectors
import pickle

from tqdm import tqdm
import os
from sklearn.feature_extraction.text import TfidfTransformer #TF-IDF is a statistical measure that evaluates how relevant a word is to a document in a collection of documents. 
from sklearn.feature_extraction.text import TfidfVectorizer  #Convert a collection of raw documents to a matrix of TF-IDF features.
from sklearn.feature_extraction.text import CountVectorizer  #Scikit-learn's CountVectorizer is used to convert a collection of text documents to a vector of term/token counts.
df = pd.read_csv('Reviews.csv')
df.head()
# check data types and counts for each column
df.info()
# check size of dataframe
df.shape
# obtain list of unique words in Summary
total_options = df["Summary"]
total_options.unique()
# obtain list of unique words in Text
total_options = df["Text"]
total_options.unique()
plt.figure(figsize=(10,5))
sns.countplot(dataframe['Score'], palette='gist_rainbow')
plt.title("Distribution of Ratings across the whole dataset")
plt.xlabel("Reviews Ratings ")
plt.ylabel("Number of reviews corresponding to the 5 ratings")
plt.show()

print(dataframe['Score'].value_counts())
#The Natural Language Toolkit (NLTK) is a platform used for building Python programs that work 
#with human language data for applying in statistical natural language processing (NLP). It contains text processing libraries 
#for tokenization, parsing, classification, stemming, tagging and semantic reasoning.
import nltk
from nltk.corpus import stopwords
# assign reviews with score > 5 as positive sentiment
# score < 5 negative sentiment
# remove score = 5
df = df[df['Score'] != 5]
df['sentiment'] = df['Score'].apply(lambda rating : +1 if rating > 5 else -1)
df.head()
!pip install wordcloud
from wordcloud import WordCloud<jupyter_output><empty_output><jupyter_text>The dataset is available in two forms1.csv file2.SQLite DatabaseIn order to load the data, I have used the SQLITE dataset as it easier to query the data and visualise the data and results efficiently.  WordcloudWord Cloud is a data visualization technique used for representing text data in which the size of each word indicates its frequency or importance. Significant textual data points can be highlighted using a word cloud.<jupyter_code>{"stream_name":"stdout","time":"0:00:00.233244","data":"Loading required package: DBI\n"},{"stream_name":"stdout","time":"0:00:00.237744","data":"Loading required package: methods\n"},{"stream_name":"stdout","time":"0:00:00.641376","data":"Loading required package: NLP\n"},{"stream_name":"stdout","time":"0:00:00.783783","data":"Loading required package: RColorBrewer\n"},{"stream_name":"stdout","time":"0:00:34.382261","data":"null device \n"},{"stream_name":"stdout","time":"0:00:34.382390","data":"          1 \n"},
from wordcloud import WordCloud, STOPWORDS
stopwords = set(STOPWORDS)

df2=final_data

plt.rcParams['figure.figsize']=(8.0,6.0)    #(6.0,4.0)
figure(num=None, figsize=(12, 10), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams['font.size']=12                #10 
plt.rcParams['savefig.dpi']=100             #72 
plt.rcParams['figure.subplot.bottom']=.1 


def show_wordcloud(data, title = None):
    wordcloud = WordCloud(
        background_color='white',
        stopwords=stopwords,
        max_words=200,
        max_font_size=40, 
        scale=3,
        random_state=1 # chosen at random by flipping a coin; it was heads
    ).generate(str(data))
    
    fig = plt.figure(1, figsize=(8, 8))
    plt.axis('off')
    if title: 
        fig.suptitle(title, fontsize=20)
        fig.subplots_adjust(top=2.3)

    plt.imshow(wordcloud)
    plt.show()
    
show_wordcloud(df2['CleanedText'])
df2.loc[df2['SentimentPolarity'] == 'Positive']['CleanedText']<jupyter_output><empty_output><jupyter_text>Reading/Selecting Data<jupyter_code># using the SQLite Table to read data.
con = sqlite3.connect('database.sqlite') 

#filtering only positive and negative reviews i.e. 
# not taking into consideration those reviews with Score=3 or regarding them as neutral
# SELECT * FROM Reviews WHERE Score != 3 LIMIT 500000, will give top 500000 data points
# you can change the number to any other number based on your computing power

filtered_data = pd.read_sql_query(""" SELECT * FROM Reviews WHERE Score != 3 LIMIT 5000""", con) 

def partition(x):
    if x < 3:
        return 0
    return 1

#changing reviews with score less than 3 to be positive and vice-versa
actualScore = filtered_data['Score']
positiveNegative = actualScore.map(partition) 
filtered_data['Score'] = positiveNegative
print("Number of data points in our data", filtered_data.shape)
filtered_data.head()<jupyter_output><empty_output><jupyter_text>Data Cleaning: Deleting duplicate valuesIt is observed that the reviews data have many duplicate entries. Hence it was necessary to remove duplicates in order to get unbiased results for the analysis of the data.Helpfulness Numerator: Number of Peoples who found the review helpful to them.Helpfulness Denominator: Number of Peoples indicated whether they found the review helpful or not.<jupyter_code>evaluate = pd.read_sql_query("""
SELECT *
FROM Reviews
WHERE Score != 3 AND UserId="AR5J8UI46CURR"
ORDER BY ProductID
""", con)
evaluate.head()
from IPython.display import Image
from IPython.core.display import HTML 
Image(url= "https://miro.medium.com/max/756/1*EyU641UXrBiTrdf10OlORg.png", width=600, height=600)


from IPython.display import Image
from IPython.core.display import HTML 
Image(url= "https://miro.medium.com/max/756/1*qO48JZ3MamjxmusG3Qot9w.png", width=600, height=600)<jupyter_output><empty_output><jupyter_text>We can see that in these two images the brand name brand is same for the products, only flavor is different.so we have to make a code where the method ensures that there is only one representative for each product and deduplication without sorting would lead to possibility of different representatives still existing for the same product. we can observe above that the same user has multiple reviews of the product with the same values for HelpfulnessDenominator,HelpfulnessNumerator, Score, Time, Text and Summary and on doing analysis it was found thatProductId(B000HDOPZG),(B000PAQ75C) was Loacker Quadratini Vanilla Wafer Cookies(Pack of 8) and so on In order to reduce redundancy and make it les complex it was decided to eliminate the rows having same parameters(except ProductId).<jupyter_code>#Sorting data according to ProductId in ascending order
sorted_data=filtered_data.sort_values('ProductId', axis=0, ascending=True, inplace=False, kind='quicksort', na_position='last')
#Deduplication of entries
final_result=sorted_data.drop_duplicates(subset={"UserId","ProfileName","Time","Text"}, keep='first', inplace=False)
final_result.shape
#how much % of data still remains
(final_result['Id'].size*1.0)/(filtered_data['Id'].size*1.0)*100
final_result = final_result[final_result.HelpfulnessNumerator<=final_result.HelpfulnessDenominator]  ## the value of HelpfulnessNumerator is greater than HelpfulnessDenominator which is not possible hence these rows are removed from calcualtions.
#the number of entries left
print(final_result.shape)

#How many positive and negative reviews are present 
final_result['Score'].value_counts()
stop=set(stopwords.words('english'))#set of stop words
sno=nltk.stem.SnowballStemmer('english') #set of snow ball stemmers in english

def cleanhtml(sentence): #function to clean html tags in a sentence
    cleannr=re.compile('<.*?>')
    comptext=re.sub(cleannr,'',sentence)
    return comptext

def cleanpunc(sentence) : #function to clean punctuation in the sentence
    cleaned=re.sub(r'[? | ! | \' |" | #]',r'',sentence)
    cleaned=re.sub(r'[. | , | ) | ( | \ | / ]' ,r' ',sentence)
    return cleaned

print(stop)
print('**' * 50)
print(sno.stem('taste'))
#We will check the distribution of stemmed word lengths across the whole review dataset to understand what is the length of the maximum number of words we will consider for the word to be relevant.
#In other words we will keep only those words which has a length less than that of a speicific length (we will obtain this specific length from the histogram).
total_words = []

for review in tqdm(final_data['Text'].values):
    filtered_sentence=[]
    review = decontracted(review)
    review = removeNumbers(review)
    review = removeHtml(review)
    review = removeURL(review)
    review = removePunctuations(review)
    review = removePatterns(review)
    
    for cleaned_words in review.split():   
        if((cleaned_words not in custom_stopwords)):          
            stemed_word=(sno.stem(cleaned_words.lower()))
            total_words.append(stemed_word)

total_words = list(set(total_words)) #Get list of unique words.

#A list to hold the length of each words used in all the reviews used across the whole dataset.
dist = []
for i in tqdm(total_words):
    length = len(i)
    dist.append(length)

# matplotlib histogram to see the distribution of the length of words
plt.figure(figsize=(20,10))
plt.hist(dist, color = 'red', edgecolor = 'blue', bins =90)
plt.title('Distribution of the length of Words across all reviews.')
plt.xlabel('Word Lengths')
plt.ylabel('Number of Words')
# remove urls from text python: https://stackoverflow.com/a/40823105/4084039
# printing some random reviews
sent_0 = final_result['Text'].values[0]
print(sent_0)
print("="*50)

sent_1000 = final_result['Text'].values[1000]
print(sent_1000)
print("="*50)

sent_1400 = final_result['Text'].values[1400]
print(sent_1400)
print("="*50)

sent_4900 = final_result['Text'].values[4900]
print(sent_4900)
print("="*50)
# https://stackoverflow.com/a/47091490/4084039
import re

def deactivate(phrase):
    # specific
    phrase = re.sub(r"won't", "will not", phrase)
    phrase = re.sub(r"can\'t", "can not", phrase)

    # general
    phrase = re.sub(r"n\'t", " not", phrase)
    phrase = re.sub(r"\'re", " are", phrase)
    phrase = re.sub(r"\'s", " is", phrase)
    phrase = re.sub(r"\'d", " would", phrase)
    phrase = re.sub(r"\'ll", " will", phrase)
    phrase = re.sub(r"\'t", " not", phrase)
    phrase = re.sub(r"\'ve", " have", phrase)
    phrase = re.sub(r"\'m", " am", phrase)
    return phrase
# https://stackoverflow.com/questions/16206380/python-beautifulsoup-how-to-remove-all-tags-from-an-element
from bs4 import BeautifulSoup

soup = BeautifulSoup(sent_0, 'lxml')
text = soup.get_text()
print(text)
print("="*50)

soup = BeautifulSoup(sent_1000, 'lxml')
text = soup.get_text()
print(text)
print("="*50)

soup = BeautifulSoup(sent_1400, 'lxml')
text = soup.get_text()
print(text)
print("="*50)

soup = BeautifulSoup(sent_4900, 'lxml')
text = soup.get_text()
print(text)
sent_1400 = deactivate(sent_1400)
print(sent_1400)
print("="*50)

#remove words with numbers python: https://stackoverflow.com/a/18082370/4084039
sent_1400 = re.sub("\S*\d\S*", "", sent_1400).strip()
print(sent_1400)
#remove spacial character: https://stackoverflow.com/a/5843547/4084039
sent_1400 = re.sub('[^A-Za-z0-9]+', ' ', sent_1400)
print(sent_1400)
# https://gist.github.com/sebleier/554280
# we are removing the words from the stop words list: 'no', 'nor', 'not'
# <br /><br /> ==> after the above steps, we are getting "br br"
# we are including them into stop words list
# instead of <br /> if we have <br/> these tags would have revmoved in the 1st step

stopwords= set(['br', 'the', 'i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're", "you've",\
            "you'll", "you'd", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', \
            'she', "she's", 'her', 'hers', 'herself', 'it', "it's", 'its', 'itself', 'they', 'them', 'their',\
            'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', "that'll", 'these', 'those', \
            'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', \
            'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', \
            'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after',\
            'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further',\
            'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more',\
            'most', 'other', 'some', 'such', 'only', 'own', 'same', 'so', 'than', 'too', 'very', \
            's', 't', 'can', 'will', 'just', 'don', "don't", 'should', "should've", 'now', 'd', 'll', 'm', 'o', 're', \
            've', 'y', 'ain', 'aren', "aren't", 'couldn', "couldn't", 'didn', "didn't", 'doesn', "doesn't", 'hadn',\
            "hadn't", 'hasn', "hasn't", 'haven', "haven't", 'isn', "isn't", 'ma', 'mightn', "mightn't", 'mustn',\
            "mustn't", 'needn', "needn't", 'shan', "shan't", 'shouldn', "shouldn't", 'wasn', "wasn't", 'weren', "weren't", \
            'won', "won't", 'wouldn', "wouldn't"])

#code for step by step implementing text preprocess
#this code is takes time because it needs to run on 500 k sentences.
## Combining all the above stundents 
from tqdm import tqdm # tqdm is for printing the status bar
i=0
str1=' '
preprocessed_reviews=[]
all_positive_words=[] #store words from +ve reviews here
all_negative_words=[] # store words from -ve reviews here
s=''

for sentance in tqdm(final_result['Text'].values):
    filtered_sentence=[]
    #print(sentance)
    sentance=cleanhtml(sentance)#removing the html tags
    sentance = re.sub(r"http\S+", "", sentance)
    sentance = BeautifulSoup(sentance, 'lxml').get_text()
    sentance = deactivate(sentance)
    sentance = re.sub("\S*\d\S*", "", sentance).strip()
    sentance = re.sub('[^A-Za-z]+', ' ', sentance)
    # https://gist.github.com/sebleier/554280
    # sentance = ' '.join(e.lower() for e in sentance.split() if e.lower() not in stopwords)
    for w in sentance.split():
        for cleaned_words in cleanpunc(w).split():
            if((cleaned_words.isalpha()) & (len(cleaned_words) > 2)):
                if (cleaned_words.lower() not in stopwords):
                    s=(sno.stem(cleaned_words.lower())).encode('utf8')
                    filtered_sentence.append(s)
                    if (final_result['Score'].values[i]) == 1 :
                      all_positive_words.append(s) #list all the positive words
                    if (final_result['Score'].values[i]) == 0 :
                       all_negative_words.append(s) #list all the negative words
                else:
                    continue
            else:
                continue
    #print(filtered_sentence)
    str1=b" ".join(filtered_sentence) #final string of the filtered sentence
    
    preprocessed_reviews.append(str1.strip())
    i+=1

final_result['preprocessed_reviews']=preprocessed_reviews #adding cleaned text to the
final_result['preprocessed_reviews']=final_result['preprocessed_reviews'].str.decode("utf-8")
final_result['preprocessed_reviews'].iloc[0]
final_result.to_csv('preprocessed_reviews',index=False)
#conn=sqlite3.connect('final.sqlite')
#c=conn.cursor()
#conn.text_factory=str
#final.to_sql('pre_text',conn,schema=None,if_exists='replace')
#code for step by step implementing text preprocess
#this code is takes time because it needs to run on 50000 sentences.
from tqdm import tqdm # tqdm is for printing the status bar
i=0
str1=' '
preprocessed_summary=[]
all_positive_summary=[] #store words from +ve reviews here
all_negative_summary=[] # store words from -ve reviews here
s=''

for sentance in tqdm(final_result['Summary'].values):
    filtered_summary=[]
    #print(sentance)
    sentance=cleanhtml(sentance)#removing the html tags
    sentance = re.sub(r"http\S+", "", sentance)
    sentance = BeautifulSoup(sentance, 'lxml').get_text()
    sentance = deactivate(sentance)
    sentance = re.sub("\S*\d\S*", "", sentance).strip()
    sentance = re.sub('[^A-Za-z]+', ' ', sentance)
    # https://gist.github.com/sebleier/554280
    # sentance = ' '.join(e.lower() for e in sentance.split() if e.lower() not in stopwords)
    for w in sentance.split():
        for cleaned_words in cleanpunc(w).split():
            if((cleaned_words.isalpha()) & (len(cleaned_words) > 2)):
                if (cleaned_words.lower() not in stopwords):
                    s=(sno.stem(cleaned_words.lower())).encode('utf8')
                    filtered_summary.append(s)
                    if (final_result['Score'].values[i]) == 1 :
                      all_positive_summary.append(s) #list all the positive words
                    if (final_result['Score'].values[i]) == 0 :
                      all_negative_summary.append(s) #list all the negative words
                else:
                    continue
            else:
                continue
    #print(filtered_sentence)
    str1=b" ".join(filtered_summary) #final string of the filtered sentence
    
    preprocessed_summary.append(str1.strip())
    i+=1
final_result['preprocessed_summary']=preprocessed_summary #adding cleaned text to the
final_result['preprocessed_summary']=final_result['preprocessed_summary'].str.decode("utf-8")
final_result['preprocessed_summary'].iloc[0]
final_result.to_csv('preprocessed_data.csv',index=False)
#store all in to database for future
#conn=sqlite3.connect('preprocessed.sqlite')
#c=conn.cursor()
#conn.text_factory=str
#final.to_sql('preprocessed_data',conn,schema=None,if_exists='replace')
all_positive_summary[0]
all_positive_words[0:10]
all_negative_words[0:10]

all_negative_summary[0:10]

df.Score.value_counts().sort_index().plot.bar(alpha=0.7, grid=True, color = 'orange', width = 0.9)
plt.xlabel('Score')
plt.ylabel('Number Of Reviews')
plt.title('Distribution of reviews over each score')
plt.show()<jupyter_output><empty_output><jupyter_text>It is clear from the plot that more than 80000 reviews have given 4 score and around 30000 have given 2 scores.<jupyter_code># How many empty length texts are present in the dataset()
df[df['Text']==0].Text.count()<jupyter_output><empty_output><jupyter_text>this means there are no empty rows of text content<jupyter_code>import re
from nltk.corpus import stopwords
stopwords_en = set(stopwords.words('english'))
ps = PorterStemmer()
def clean_text(text):
    text = text.lower() #converting text to lowercase
    text = ' '.join([i for i in nltk.word_tokenize(text) if i not in stopwords_en and i not in string.punctuation]) #stopword and punct removal
    text = re.sub('[^a-z]+', ' ', text) #removal of anything other than English letters
    text = ' '.join([ps.stem(i) for i in nltk.word_tokenize(text)]) #stemming
    return text<jupyter_output><empty_output><jupyter_text>Data Preprocessing Here we will apply all the basic steps that I have discussed above and finally we collect the words used to describe positive and negative reviews<jupyter_code># printing some random reviews
sent_0 = final_result['Text'].values[0]
print(sent_0)
print("="*50)

sent_1001 = final_result['Text'].values[1000]
print(sent_1001)
print("="*50)

sent_1400 = final_result['Text'].values[1500]
print(sent_1400)
print("="*50)

sent_4560 = final_result['Text'].values[4900]
print(sent_4560)
print("="*50)
import re

def deactivate(phrase):
    # specific
    phrase = re.sub(r"won't", "will not", phrase)
    phrase = re.sub(r"can\'t", "can not", phrase)

    # general
    phrase = re.sub(r"n\'t", " not", phrase)
    phrase = re.sub(r"\'re", " are", phrase)
    phrase = re.sub(r"\'s", " is", phrase)
    phrase = re.sub(r"\'d", " would", phrase)
    phrase = re.sub(r"\'ll", " will", phrase)
    phrase = re.sub(r"\'t", " not", phrase)
    phrase = re.sub(r"\'ve", " have", phrase)
    phrase = re.sub(r"\'m", " am", phrase)
    return phrase
# we are removing the words from the stop words list: 'no', 'nor', 'not'
# <br /><br /> ==> after the above steps, we are getting "br br"
# we are including them into stop words list
# instead of <br /> if we have <br/> these tags would have removed in the 1st step

stopwords= set(['br', 'the', 'i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're", "you've",\
            "you'll", "you'd", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', \
            'she', "she's", 'her', 'hers', 'herself', 'it', "it's", 'its', 'itself', 'they', 'them', 'their',\
            'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', "that'll", 'these', 'those', \
            'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', \
            'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', \
            'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after',\
            'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further',\
            'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more',\
            'most', 'other', 'some', 'such', 'only', 'own', 'same', 'so', 'than', 'too', 'very', \
            's', 't', 'can', 'will', 'just', 'don', "don't", 'should', "should've", 'now', 'd', 'll', 'm', 'o', 're', \
            've', 'y', 'ain', 'aren', "aren't", 'couldn', "couldn't", 'didn', "didn't", 'doesn', "doesn't", 'hadn',\
            "hadn't", 'hasn', "hasn't", 'haven', "haven't", 'isn', "isn't", 'ma', 'mightn', "mightn't", 'mustn',\
            "mustn't", 'needn', "needn't", 'shan', "shan't", 'shouldn', "shouldn't", 'wasn', "wasn't", 'weren', "weren't", \
            'won', "won't", 'wouldn', "wouldn't"])
# Data cleaning for all the reviews 
from bs4 import BeautifulSoup   #Beautiful Soup is a Python library for pulling data out of HTML and XML files. It works with your favorite parser to provide idiomatic ways of navigating, searching, and modifying the parse tree.
                                #It commonly saves programmers hours or days of work.

from tqdm import tqdm
pp_reviews = []
# tqdm is for printing the status bar
for sentence in tqdm(final_result['Text'].values):
    sentence = re.sub(r"http\S+", "", sentence)           #remove urls
    sentence = BeautifulSoup(sentence, 'lxml').get_text() #remove xml tags
    sentence = deactivate(sentence)                     
    sentence = re.sub("\S*\d\S*", "", sentence).strip()   #remove words with numbers
    sentence = re.sub('[^A-Za-z]+', ' ', sentence)        #remove special characters
    sentence = ' '.join(e.lower() for e in sentence.split() if e.lower() not in stopwords)
    pp_reviews.append(sentence.strip())
pp_reviews[0]

len(pp_reviews)<jupyter_output><empty_output><jupyter_text>Featurization<jupyter_code>from numpy import zeros
import keras
import tensorflow as tf
from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences
from keras.models import Sequential
from keras.callbacks import EarlyStopping
from keras.layers.embeddings import Embedding
from keras.layers import SpatialDropout1D
# prepare tokenizer
t = Tokenizer()
t.fit_on_texts(pp_reviews)
vocab_size = len(t.word_index) + 1
# integer encode the documents
code_docs = t.texts_to_sequences(pp_reviews)
# pad documents to a max length of max words
max_length = 150
pad_docs = pad_sequences(code_docs, maxlen=max_length, padding='post')
label = final_result['Score']
pad_docs[1500]<jupyter_output><empty_output><jupyter_text>Prepare Model<jupyter_code>import re
from nltk.corpus import stopwords
stopwords_en = set(stopwords.words('english'))
ps = PorterStemmer()
def clean_text(text):
    text = text.lower() #converting text to lowercase
    text = ' '.join([i for i in nltk.word_tokenize(text) if i not in stopwords_en and i not in string.punctuation]) #stopword and punct removal
    text = re.sub('[^a-z]+', ' ', text) #removal of anything other than English letters
    text = ' '.join([ps.stem(i) for i in nltk.word_tokenize(text)]) #stemming
    return text
# Apply the cleanup and create a new column
df['FilterText'] = df['Text'].apply(lambda x: clean_text(x))
df.head(3)
def partition(val):
    if(val>2):
        return 1
    return 0
df['Positivity']=df['Score'].apply(lambda x: partition(x))
df.head(3)
required_columns = ['FilterText', 'Positivity']
df = df[required_columns]
df.head()
df.Positivity.value_counts().plot.bar(alpha=0.5, grid=True)
plt.title('Distribution Of Positive & Negative Feedback')
plt.ylabel('Counts')
plt.show()<jupyter_output><empty_output><jupyter_text>Around 120000 people have given positive reviews(1) and around 80000 people have given negative reviews.<jupyter_code># Create Train/Test data split model
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(df['FilterText'], df['Positivity'], test_size=0.25, random_state=42, shuffle=True, stratify=df['Positivity'])
print("Train Set Size = {}\nTest Set Size  = {}".format(X_train.shape[0], X_test.shape[0]))<jupyter_output><empty_output><jupyter_text>Long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture used in the field of deep learning. LSTM networks are well-suited to classifying, processing and making predictions based on time series data, since there can be lags of unknown duration between important events in a time series.<jupyter_code>#BoW
count_vect = CountVectorizer() #in scikit-learn
count_vect.fit(preprocessed_reviews)
print("some feature names ", count_vect.get_feature_names()[500:510])
print('='*50)

final_counts = count_vect.transform(pp_reviews)
print("the type of count vectorizer ",type(final_counts))
print("the shape of out text BOW vectorizer ",final_counts.get_shape())
print("the number of unique words ", final_counts.get_shape()[1])
data=pd.read_csv('preprocessed_data.csv')
data.head()
data['preprocessed_reviews'].isnull().sum()
#bi-gram, tri-gram and n-gram
#removing stop words like "not" should be avoided before building n-grams
# count_vect = CountVectorizer(ngram_range=(1,2))
# you can choose these numbers min_df=10, max_features=5000
count_vect = CountVectorizer(ngram_range=(1,2), min_df=10, max_features=5000)
final_bigram_counts = count_vect.fit_transform(data['preprocessed_reviews'])[:5000]
print("the type of count vectorizer ",type(final_bigram_counts))
print("the shape of out text BOW vectorizer ",final_bigram_counts.get_shape())
print("the number of unique words including both unigrams and bigrams ", final_bigram_counts.get_shape()[1])<jupyter_output><empty_output><jupyter_text>TF-IDF(Term frequency -inverse document frequency):    TF-IDF stands for “Term Frequency — Inverse Document Frequency”. This is a technique to quantify a word in documents, we generally compute a weight to each word which signifies the importance of the word in the document and corpus. This method is a widely used technique in Information Retrieval and Text Mining.<jupyter_code>tf_idf_vect = TfidfVectorizer(ngram_range=(1,2), min_df=10)
tf_idf_vect.fit(data['preprocessed_reviews'])
print("some sample features(unique words in the corpus)",tf_idf_vect.get_feature_names()[0:10])
print('='*50)

final_tf_idf = tf_idf_vect.transform(data['preprocessed_reviews'])
print("the type of count vectorizer ",type(final_tf_idf))
print("the shape of out text TFIDF vectorizer ",final_tf_idf.get_shape())
print("the number of unique words including both unigrams and bigrams ", final_tf_idf.get_shape()[1])<jupyter_output><empty_output><jupyter_text>word2vec:This technique is the state of the art algorithm, it consider the semantic meaning of the word.If we give the word it converts in to vectors. It also learns relationship automatically from the text.The output of the word2vec model is Dense vectors.Word2vec model requires large text corpus.<jupyter_code>data=pd.read_csv('preprocessed_data.csv')
data['preprocessed_reviews']=data['preprocessed_reviews'].fillna(method='bfill')
data['preprocessed_reviews'].isnull().sum()
# Train your own Word2Vec model using your own text corpus
i=0
list_of_sentance=[]
for sentance in data['preprocessed_reviews']:
    list_of_sentance.append(sentance.split())
list_of_sentance[0]
# Using Google News Word2Vectors
# in this project we are using a pretrained model by google 
#  a pickle file wich contains a dict , 
# and it contains all our courpus words as keys and  model[word] as values
# To use this code-snippet, download "GoogleNews-vectors-negative300.bin" 
# from https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit

is_your_ram_gt_16g=False
want_to_use_google_w2v = False
want_to_train_w2v = True

if want_to_train_w2v:
    # min_count = 5 considers only words that occured atleast 5 times
    w2v_model=Word2Vec(list_of_sentance,min_count=5, workers=4)
    print(w2v_model.wv.most_similar('great'))
    print('='*50)
    print(w2v_model.wv.most_similar('worst'))
    
elif want_to_use_google_w2v and is_your_ram_gt_16g:
    if os.path.isfile('GoogleNews-vectors-negative300.bin'):
        w2v_model=KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)
        print(w2v_model.wv.most_similar('great'))
        print(w2v_model.wv.most_similar('worst'))
    else:
        print("you don't have gogole's word2vec file, keep want_to_train_w2v = True, to train your own w2v ")
from sklearn.feature_extraction.text import TfidfVectorizer
tfidf_vectoriser = TfidfVectorizer()

# Training % Feature Extraction On Entire Dataset, Used For Cross Validation & Model Comparison
features = tfidf_vectoriser.fit_transform(df['FilterText'])
labels = df['Positivity'].astype(int)
# Training On Only Train Set Now
tfidf_vectoriser.fit(X_train)
X_train_tf = tfidf_vectoriser.transform(X_train)
X_test_tf = tfidf_vectoriser.transform(X_test)
X_train_tf.shape, X_test_tf.shape
import random
print("Twenty Random Words from Training Set ...\n",*random.sample(tfidf_vectoriser.get_feature_names(),20))
from sklearn.naive_bayes import MultinomialNB #The multinomial Naive Bayes classifier is suitable for classification with discrete features (e.g., word counts for text classification). The multinomial distribution normally requires integer feature counts. However, in practice, fractional counts such as tf-idf may also work. Parameters alphafloat, default=1.0.
from sklearn.svm import LinearSVC #The objective of a Linear SVC (Support Vector Classifier) is to fit to the data you provide, returning a "best fit" hyperplane that divides, or categorizes, your data
from sklearn.model_selection import cross_val_score #Cross-validation is a statistical method used to estimate the skill of machine learning models.That k-fold cross validation is a procedure used to estimate the skill of the model on new data. There are common tactics that you can use to select the value of k for your dataset.
from sklearn.linear_model import LogisticRegression #Logistic Regression is a Machine Learning classification algorithm that is used to predict the probability of a categorical dependent variable.
from sklearn.ensemble import RandomForestClassifier #A random forest classifier. A random forest is a meta estimator that fits a number of decision tree classifiers on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting.
parameters = [
    LinearSVC(),
    MultinomialNB(),
    LogisticRegression(random_state=0),
    RandomForestClassifier(n_estimators=200, n_jobs=-1, random_state=0)
]
CV = 5
cv_df = pd.DataFrame(index=range(CV * len(parameters)))
entries = []
for model in parameters:
    model_name = model.__class__.__name__
    accuracies = cross_val_score(model, features ,labels, scoring='accuracy', cv=CV)
    for fold_idx, accuracy in enumerate(accuracies):
        entries.append((model_name, fold_idx, accuracy))
from sklearn.model_selection import GridSearchCV
param_grid = {
    'C':[0.5,0.8,1.0,1.5]
}
svm = LinearSVC(max_iter=1500)
svm_cv = GridSearchCV(svm, param_grid, cv=5)
svm_cv.fit(features, labels)
print("Best Parameters :", svm_cv.best_params_)
print("Amazing Score :",svm_cv.best_score_)
svm = LinearSVC(C=0.5, max_iter=2000)
svm.fit(X_train_tf, y_train)
from sklearn.metrics import classification_report, accuracy_score, confusion_matrix, roc_auc_score, roc_curve
y_pred = svm.predict(X_test_tf)
print(classification_report(y_test, y_pred, target_names=['Negative','Positive']))
print("Accuracy :",accuracy_score(y_test, y_pred), end='\n\n')
conf_mat = confusion_matrix(y_test, y_pred)
print(conf_mat)<jupyter_output><empty_output><jupyter_text>Preprocessing the data using Spacy and Machine learning model training using sklearnIn this stage, Spacy package of python is used to lemmatize and remove stop words from the obtained dataset.<jupyter_code>from  spacy.lang.en.stop_words import STOP_WORDS
# To build a list of stop words for filtering
stopwords = list(STOP_WORDS)
print(stopwords)<jupyter_output><empty_output><jupyter_text>Thus, the stop words have been enlistedThe data is initially split into test and training datasets prior to feeding into the machine learning pipeline. Then, a class object was defined as 'sent_predict' is used as the first step of the pipeline which would inherit from the TransformerMixin package and perform the cleaning of data. The second method of the pipeline is to vectorize the cleaned data.Tokenized words needs to be lemmatized and filtered for pronouns, stopwords and punctuations using the defined method 'my_tokenizer' For that purpose count vectorizeor and tfidfVectorizer both have been tried subsequently to decide which is better.Then the third step of the pipeline is the defining of the classifier. In this case, Linear Support Vector Machine classifier was chosen. Other methods could be explored in the furture<jupyter_code>import string
punctuations = string.punctuation
# Creating a Spacy Parser
from spacy.lang.en import English
parser = English()
def my_tokenizer(sentence):
    mytokens = parser(sentence)
    mytokens = [ word.lemma_.lower().strip() if word.lemma_ != "-PRON-" else word.lower_ for word in mytokens ]
    mytokens = [ word for word in mytokens if word not in stopwords and word not in punctuations ]
    return mytokens
from sklearn.metrics import accuracy_score 
from sklearn.base import TransformerMixin 
from sklearn.pipeline import Pipeline
# Create the  pipeline to clean, tokenize, vectorize, and classify using"Count Vectorizor"
pipe_countvect = Pipeline([("cleaner", predictors()),
                 ('vectorizer', vectorizer),
                 ('classifier', classifier)])
# Fit our data
pipe_countvect.fit(X_train,y_train)
# Predicting with a test dataset
sample_prediction = pipe_countvect.predict(X_test)
#Custom transformer using spaCy 
class predictors(TransformerMixin):
    def transform(self, X, **transform_params):
        return [clean_text(text) for text in X]
    def fit(self, X, y, **fit_params):
        return self
    def get_params(self, deep=True):
        return {}

# Basic function to clean the text 
def clean_text(text):     
    return text.strip().lower()
from sklearn.feature_extraction.text import CountVectorizer,TfidfVectorizer
from sklearn.metrics import accuracy_score 
from sklearn.base import TransformerMixin 
from sklearn.pipeline import Pipeline
from sklearn.svm import LinearSVC
# Vectorization
vectorizer = CountVectorizer(tokenizer = my_tokenizer, ngram_range=(1,1)) 
classifier = LinearSVC()
# Using Tfidf
tfvectorizer = TfidfVectorizer(tokenizer = my_tokenizer)
df1 = pd.read_csv('pp_reviews.csv')
from sklearn.model_selection import train_test_split
# Features and Labels
X = df1['Text']
ylabels = df1['Score']
pipe = Pipeline([("cleaner", predictors()),
                 ('vectorizer', vectorizer),
                 ('classifier', classifier)])
pipe.fit(X_train,y_train)
sample_prediction = pipe.predict(X_test)
def removeHTMLTags(review):
    soup = BeautifulSoup(review, 'lxml')
    return soup.get_text()
def removeApostrophe(review):
    phrase = re.sub(r"won't", "will not", review)
    phrase = re.sub(r"can\'t", "can not", review)
    phrase = re.sub(r"n\'t", " not", review)
    phrase = re.sub(r"\'re", " are", review)
    phrase = re.sub(r"\'s", " is", review)
    phrase = re.sub(r"\'d", " would", review)
    phrase = re.sub(r"\'ll", " will", review)
    phrase = re.sub(r"\'t", " not", review)
    phrase = re.sub(r"\'ve", " have", review)
    phrase = re.sub(r"\'m", " am", review)
    return phrase

def removeAlphaNumericWords(review):
     return re.sub("\S*\d\S*", "", review).strip()
 
def removeSpecialChars(review):
     return re.sub('[^a-zA-Z]', ' ', review)
def doTextCleaning(review):
    review = removeHTMLTags(review)
    review = removeApostrophe(review)
    review = removeAlphaNumericWords(review)
    review = removeSpecialChars(review) 
    # Lower casing
    review = review.lower()  
    #Tokenization
    review = review.split()
    #Removing Stopwords and Lemmatization
    lmtzr = WordNetLemmatizer()
    review = [lmtzr.lemmatize(word, 'v') for word in review if not word in set(stopwords.words('english'))]
    review = " ".join(review)    
    return review
def scorePartition(x):
    if x < 3:
        return 0
    return 1
final=final_result.sort_values(by=['Time'],ascending=False)
finalDataPoints=final_result.head(100000)
x=finalDataPoints["Text"]
y=finalDataPoints["Score"]

x_tr,x_test,y_tr,y_test=train_test_split(x, y, test_size=0.2,shuffle=False)
#BOW for unigram
bow = CountVectorizer()
x_tr_uni = bow.fit_transform(x_tr)
x_test_uni= bow.transform(x_test)

from sklearn.preprocessing import StandardScaler
x_tr_uni = StandardScaler(with_mean = False).fit_transform(x_tr_uni)
x_test_uni = StandardScaler(with_mean = False).fit_transform(x_test_uni)
pip install sklearn 0.21.3
pip install best_score
LR = LogisticRegression(penalty='l2')
C_value = [{'C': [10**-4, 10**-2, 10**0, 10**2, 10**4]}]
gsv = GridSearchCV(LR,C_value,cv=5,verbose=1,scoring='f1_weighted')
gsv.fit(x_tr_uni,y_tr)
print("Best HyperParameter: ",gsv.best_params_)
print(gsv.best_score)
optimal_C=gsv.best_score

x=[]
y=[]
plt.figure(figsize=(8,8))
for a in gsv.cv_scores_:
    x.append(a[0]['C']) 
    y.append(a[1])
plt.xlabel("C",fontsize=15)
plt.ylabel("f1_weighted")
plt.title('f1_weighted v/s C')
plt.plot(x,y, marker='o', markerfacecolor='red', markersize=10)
plt.show()
#Plot the performance of model both on train data and cross validation data for each hyper parameter. Display the confusion matrix as well as ROC Curve
#From the heatmaps above, we will select the optimal value of max_depth to be 25 and min_samples_split to be 500.
max_depth = 25
min_samples_split=500,

best_estimator = DecisionTreeClassifier(criterion='gini', max_depth=25, min_samples_split=500, random_state=0, splitter='best')
vectorizationType="TFIDF" 
trained_clf_TFIDF = performance(best_estimator, vectorizationType, X_train, y_train, X_test, y_test, X_calib, y_calib, max_depth, min_samples_split)
Call the function above and pass a filename onto it.
f_names=tf_idf_obj.get_feature_names()

graph=visualize_tree(trained_clf_TFIDF, f_names, 'TFIDF_DT.png')
graph
#Load the W2V Vectors we had created earlier and standardize them. We will standardize the traina and test data seperately in order to prevent data leakage.
import pickle

with open('X_train_W2V.pkl', 'rb') as file:
    X_train = pickle.load(file)
    
with open('X_test_W2V.pkl', 'rb') as file:
    X_test = pickle.load(file)

with open('y_train_W2V.pkl', 'rb') as file:
    y_train = pickle.load(file)
    
with open('y_test_W2V.pkl', 'rb') as file:
    y_test = pickle.load(file)
    
with open('X_calib_W2V.pkl', 'rb') as file:
    X_calib = pickle.load(file)

with open('y_calib_W2V.pkl', 'rb') as file:
    y_calib = pickle.load(file)    


print("Shape of the train data matrix: ",X_train.shape)
print("Shape of the test data matrix: ",X_test.shape)
print("Shape of the calibration data matrix: ",X_calib.shape)

#Perform Grid Search cross validation to obtain the best value of the hyperparameter.
vectorizationType = "AVG-W2V"
st=datetime.now()
gsearch_cv = get_GridSearchCV(vectorizationType, X_train, y_train, X_test, y_test)
print("\nTime taken to complete Hyperparameter Search: ",datetime.now()-st)
#Plot the performance of model both on train data and cross validation data for each hyper parameter. Display the confusion matrix as well as ROC Curve
#From the heatmaps above, we will select the optimal value of max_depth to be 10 and min_samples_split to be 250.
max_depth = 10
min_samples_split=250,

best_estimator = DecisionTreeClassifier(criterion='gini', max_depth=10, min_samples_split=250, random_state=0, splitter='best')
vectorizationType="AVG-W2V" 
trained_clf_avg_w2v = performance(best_estimator, vectorizationType, X_train, y_train, X_test, y_test, X_calib, y_calib, max_depth, min_samples_split)
#Load the TF-IDF W2V Vectors we had created earlier and standardize them. We will standardize the traina and test data seperately in order to prevent data leakage.
import pickle
    
with open('X_train_TFIDF-W2V.pkl', 'rb') as file:
    X_train = pickle.load(file)
    
with open('X_test_TFIDF-W2V.pkl', 'rb') as file:
    X_test = pickle.load(file)

with open('y_train_TFIDF-W2V.pkl', 'rb') as file:
    y_train = pickle.load(file)
    
with open('y_test_TFIDF-W2V.pkl', 'rb') as file:
    y_test = pickle.load(file)
    
with open('X_calib_TFIDF-W2V.pkl', 'rb') as file:
    X_calib = pickle.load(file)

with open('y_calib_TFIDF-W2V.pkl', 'rb') as file:
    y_calib = pickle.load(file)    

print("Shape of the train data matrix: ",X_train.shape)
print("Shape of the test data matrix: ",X_test.shape)
print("Shape of the calibration data matrix: ",X_calib.shape)

#Perform Grid Search cross validation to obtain the best value of the hyperparameter.
vectorizationType = "TFIDF-W2V"
st=datetime.now()
gsearch_cv = get_GridSearchCV(vectorizationType, X_train, y_train, X_test, y_test)
print("\nTime taken to complete Hyperparameter Search: ",datetime.now()-st)
#Plot the performance of model both on train data and cross validation data for each hyper parameter. Display the confusion matrix as well as ROC Curve
#From the heatmaps above, we will select the optimal value of max_depth to be 10 and min_samples_split to be 500.
max_depth = 10
min_samples_split=500,

best_estimator = DecisionTreeClassifier(criterion='gini', max_depth=10, min_samples_split=500, random_state=0, splitter='best')
vectorizationType="TFIDF-W2V" 
trained_clf_avg_w2v = performance(best_estimator, vectorizationType, X_train, y_train, X_test, y_test, X_calib, y_calib, max_depth, min_samples_split)
Compare performance and display it on a pretty table.
from prettytable import PrettyTable
table = PrettyTable()
table.field_names = ["Model", "Max Depth", "Min Samples Split", "Accuracy on Test data", "AUC Score Test Data", "No. Of accurate predictions"]

print("Please find below the important metrics for all the models below.\n")
file = open('info_model_DT.txt', 'r')
file.seek(0)
for line in file:
    table.add_row(line.split())
print(table)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11092325125456409, 'fraction_numerical': 0.015900138496681475, 'mean_word_length': 4.975709372312983, 'pattern_counts': {'":': 18, '<': 91, '<?xml version=': 0, '>': 99, 'https://': 16, 'lorem ipsum': 0, 'www.': 1, 'xml': 9}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '658618', 'license': 'Apache-2.0', 'n_tokens_mistral': 17776, 'n_tokens_neox': 16728, 'n_words': 5772, 'path': 'sentiment analysis.ipynb', 'repo_name': 'NiiT-Kolkata/Amazon-Fine-Foods-Sentiment-Analysis', 'size': '55597'}