<jupyter_start><jupyter_text>Amazon Fine Food Reviews PreprocessingThis IPython notebook consists code for preprocessing of text, conversion of text into vectors and saving that information for further use.Data Source: https://www.kaggle.com/snap/amazon-fine-food-reviews Public Information - The Amazon Fine Food Reviews dataset consists of reviews of fine foods from Amazon.  1. Number of reviews: 568,454  2. Number of users: 256,059  3. Number of products: 74,258  4. Timespan: Oct 1999 - Oct 2012  5. Number of Attributes/Columns in data: 10 Attribute Information -  1. Id  2. ProductId - unique identifier for the product  3. UserId - unqiue identifier for the user  4. ProfileName  5. HelpfulnessNumerator - number of users who found the review helpful  6. HelpfulnessDenominator - number of users who indicated whether they found the review helpful or not  7. Score - rating between 1 and 5  8. Time - timestamp for the review  9. Summary - brief summary of the review  10. Text - text of the review Current Objective -   Go through the reviews and perform preprocessing, convert them into vectors and save them for future use.  Let's start with importing necessary libraries<jupyter_code>#importing necessary libraries
%matplotlib inline
import warnings
warnings.filterwarnings("ignore")

import sqlite3
import pandas as pd
import numpy as np
import missingno as msno
import pickle

import re
import os
#os.getcwd()
dir_path = os.path.abspath(os.path.join(os.getcwd(), '..'))

import nltk
from nltk.stem.wordnet import WordNetLemmatizer
from nltk.corpus import stopwords
from nltk import pos_tag, word_tokenize

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics import confusion_matrix
from sklearn import metrics
from sklearn.metrics import roc_curve, auc
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction.text import TfidfVectorizer

from gensim.models import KeyedVectors
#connecting to sqlite db
con = sqlite3.connect(os.path.join(dir_path, 'datasets','amzn_fine_food_reviews', 'database.sqlite'))

#filtering only positive and negative reviews
data = pd.read_sql_query("SELECT * FROM Reviews WHERE Score != 3", con)

print("Shape of data:", data.shape)

#scores < 3 are considered to be negative reviews and > 3 are considered to be positive reviews
data.head()<jupyter_output>Shape of data: (525814, 10)<jupyter_text>Missing values? Yeah, it happens...<jupyter_code>#let's just check, just in case if any
print("Missing values? Ans -", data.isnull().values.any())

#visualizing it
msno.matrix(data, figsize=(15,7))<jupyter_output>Missing values? Ans - False<jupyter_text>One can write more than one review for the same product!<jupyter_code>df = data.copy()
df['ProdUser'] = df['ProductId'] + df['UserId']
df[df['ProdUser'].duplicated(keep=False)].sort_values('ProdUser', axis=0, ascending=True, inplace=False, kind='quicksort', na_position='last')<jupyter_output><empty_output><jupyter_text>Obeservations1.   There are some instances where a user has written more than one review for the same product.2.   We can remove the one which has less Helpfulness but lets keep all and treat it as review from a different user.3.   Will definitely have to remove same reviews because it is just redundant data.<jupyter_code>#Sorting data according to ProductId in ascending order
data = data.sort_values('ProductId', axis=0, ascending=True, inplace=False, kind='quicksort', na_position='last')
#print("##")
#Deduplication of entries (364173, 10)
data=data.drop_duplicates(subset={"UserId","ProfileName","Time","Text"}, keep='first', inplace=False)
data.shape
data.head(2)<jupyter_output><empty_output><jupyter_text>Same reviews on multiple products with different timestamps!!<jupyter_code>data[data['Text'].duplicated(keep=False)].sort_values('Text', axis=0, ascending=True, inplace=False, kind='quicksort', na_position='last')
#removing duplicate reviews
data=data.drop_duplicates(subset={"Text"}, keep='first', inplace=False)
data.shape<jupyter_output><empty_output><jupyter_text>Observations1.   There are reviews which are same on similar products (mostly different flavors). 2.   These reviews were posted with different timestamps by the same person (weird).3.   Since we are interested in a review being positive or negative, having redundant reviews makes no sense, so removing them.<jupyter_code>#also removing those reviews where HelpfulnessNumerator is greater than HelpfulnessDenominator which is not possible
data=data[data['HelpfulnessNumerator']<=data['HelpfulnessDenominator']]
data.shape<jupyter_output><empty_output><jupyter_text>Now, it's time for some Text PreprocessingWe will be doing the following in order.1.   Text cleaning - includes removal of special characters which are not required.2.   Check if the word is actually an English word.3.   Check to see if the length of the word is greater than 2 (as it was researched that there is no adjective in 2-letters)4.   Convert the word to lower case.5.   Remove stop words but let's keep words like 'not' which makes the sentence negative.6.   POS Tagging and WordNet Lemmatizing the word.<jupyter_code>def cleanhtml(sentence): #function to clean the word of any html-tags
    cleanr = re.compile('<.*?>')
    cleantext = re.sub(cleanr, ' ', sentence)
    return cleantext
def cleanpunc(sentence): #function to clean the word of any punctuation or special characters
    cleaned = re.sub(r'[?|!|\'|"|#]',r'',sentence)
    cleaned = re.sub(r'[.|,|)|(|\|/]',r' ',cleaned)
    return  cleaned
stop = list(set(stopwords.words('english'))) #set of stopwords
print(stop)

#removing words like 'not' that gives negative meaning to a sentence from stopwords
important_stopwords = ['hadn', 'weren', 'shouldn', "needn't", 'needn', 'doesn', "shan't", "shouldn't", "wasn", "couldn't", 'mustn', "hadn't", "doesn't",
                      'wouldn', "weren't", "didn", "mustn't", "wasn't", "didn't", "don't", "not"]
pre_final_stops = [x for x in stop if x not in important_stopwords]

#removing punctuation from stop words
final_stops = list(set([cleanpunc(x) for x in pre_final_stops]))
print("Final stopwords:", final_stops)
#creating word net lemmatizer object
wnl = WordNetLemmatizer()
#Code for implementing step-by-step the checks mentioned in the pre-processing phase
# this code takes a while to run as it needs to run on 500k sentences.
i=0
str1=' '
final_string=[]
all_positive_words=[] # store words from +ve reviews here
all_negative_words=[] # store words from -ve reviews here.
s=''
scores = data['Score'].values
for sent in data['Text'].values:
    filtered_sentence=[]
    #print(sent);
    sent=cleanhtml(sent) # remove HTMl tags
    tokens = pos_tag(word_tokenize(sent))
    for w in tokens:
        for cleaned_words in cleanpunc(w[0]).split():
            if((cleaned_words.isalpha()) & (len(cleaned_words)>2)):    
                if(cleaned_words.lower() not in stop):
                    #s=(sno.stem(cleaned_words.lower())).encode('utf8')
                    # lemmatization works better with POS tagging
                    tag = w[1][0].lower()
                    tag = tag if tag in ['a', 'n', 'v'] else None
                    if not tag:
                        s = cleaned_words.lower().encode('utf8')
                    else:
                        s = wnl.lemmatize(cleaned_words.lower(), tag).lower().encode("utf8")
                    filtered_sentence.append(s)
                    if scores[i] > 3: 
                        all_positive_words.append(s) #list of all words used to describe positive reviews
                    if scores[i] < 3:
                        all_negative_words.append(s) #list of all words used to describe negative reviews reviews
                else:
                    continue
            else:
                continue 
    #print(filtered_sentence)
    str1 = b" ".join(filtered_sentence) #final string of cleaned words
    #print("***********************************************************************")
    
    final_string.append(str1)
    i+=1
print("Done!")
data['CleanedText']=final_string #adding a column of CleanedText which displays the data after pre-processing of the review 
data['CleanedText']=data['CleanedText'].str.decode("utf-8")
# store final table into an SQlLite table for future.
conn = sqlite3.connect(os.path.join(dir_path, 'datasets','amzn_fine_food_reviews', 'final.sqlite'))
c=conn.cursor()
conn.text_factory = str
data.to_sql('Reviews', conn,  schema=None, if_exists='replace', index=True, index_label=None, chunksize=None, dtype=None)
#BoW
count_vect = CountVectorizer() #in scikit-learn
final_counts = count_vect.fit_transform(data['CleanedText'].values)
print("the type of count vectorizer ",type(final_counts))
print("the shape of out text BOW vectorizer ",final_counts.get_shape())
print("the number of unique words ", final_counts.get_shape()[1])
freq_dist_positive=nltk.FreqDist(all_positive_words)
freq_dist_negative=nltk.FreqDist(all_negative_words)
print("Most Common Positive Words : ",freq_dist_positive.most_common(20))
print("Most Common Negative Words : ",freq_dist_negative.most_common(20))
#bi-gram, tri-gram and n-gram

#removing stop words like "not" should be avoided before building n-grams
count_vect = CountVectorizer(ngram_range=(1,2) ) #in scikit-learn
final_bigram_counts = count_vect.fit_transform(data['CleanedText'].values)
print("the type of count vectorizer ",type(final_bigram_counts))
print("the shape of out text BOW vectorizer ",final_bigram_counts.get_shape())
print("the number of unique words including both unigrams and bigrams ", final_bigram_counts.get_shape()[1])
tf_idf_vect = TfidfVectorizer(ngram_range=(1,2))
final_tf_idf = tf_idf_vect.fit_transform(data['CleanedText'].values)
print("the type of count vectorizer ",type(final_tf_idf))
print("the shape of out text TFIDF vectorizer ",final_tf_idf.get_shape())
print("the number of unique words including both unigrams and bigrams ", final_tf_idf.get_shape()[1])
features = tf_idf_vect.get_feature_names()
print("some sample features(unique words in the corpus)",features[100000:100010])
def top_tfidf_feats(row, features, top_n=25):
    ''' Get top n tfidf values in row and return them with their corresponding feature names.'''
    topn_ids = np.argsort(row)[::-1][:top_n]
    top_feats = [(features[i], row[i]) for i in topn_ids]
    df = pd.DataFrame(top_feats)
    df.columns = ['feature', 'tfidf']
    return df

top_tfidf = top_tfidf_feats(final_tf_idf[1,:].toarray()[0],features,25)
top_tfidf
con = sqlite3.connect(os.path.join(dir_path, 'datasets','amzn_fine_food_reviews','final.sqlite'))
data = pd.read_sql_query(""" SELECT * FROM Reviews """, con)
del data['index']
data.shape
from gensim.models import Word2Vec
w2v_model=Word2Vec(list_of_sent,min_count=5,size=50, workers=7)
#saving w2v model
w2v_model.save(os.path.join(dir_path, 'datasets','amzn_fine_food_reviews',"amzn_w2v_vec.model"))
#loading model
w2v_model = Word2Vec.load(os.path.join(dir_path, 'datasets','amzn_fine_food_reviews',"amzn_w2v_vec.model"))
list_of_sent=[]
for sent in data['CleanedText'].values:
    list_of_sent.append(sent.split())
w2v_words = list(w2v_model.wv.vocab)
print("number of words that occured minimum 5 times ",len(w2v_words))
print("sample words ", w2v_words[0:50])
len(list_of_sent)
from concurrent.futures import ThreadPoolExecutor
from concurrent import futures
# average Word2Vec
# compute average word2vec for each review.
sent_vectors = []; # the avg-w2v for each sentence/review is stored in this list
for sent in list_of_sent: # for each review/sentence
    sent_vec = np.zeros(50) # as word vectors are of zero length
    cnt_words =0; # num of words with a valid vector in the sentence/review
    for word in sent: # for each word in a review/sentence
        if word in w2v_words:
            vec = w2v_model.wv[word]
            sent_vec += vec
            cnt_words += 1
    if cnt_words != 0:
        sent_vec /= cnt_words
    sent_vectors.append(sent_vec)
print(len(sent_vectors))
print(len(sent_vectors[0]))
# TF-IDF weighted Word2Vec
tfidf_feat = tf_idf_vect.get_feature_names() # tfidf words/col-names
# final_tf_idf is the sparse matrix with row= sentence, col=word and cell_val = tfidf

tfidf_sent_vectors = []; # the tfidf-w2v for each sentence/review is stored in this list
row=0;
for sent in list_of_sent: # for each review/sentence 
    sent_vec = np.zeros(50) # as word vectors are of zero length
    weight_sum =0; # num of words with a valid vector in the sentence/review
    for word in sent: # for each word in a review/sentence
#         if word in w2v_words:
        try:
            vec = w2v_model.wv[word]
            # obtain the tf_idfidf of a word in a sentence/review
            tf_idf = final_tf_idf[row, tfidf_feat.index(word)]
            sent_vec += (vec * tf_idf)
            weight_sum += tf_idf
        except:
            pass
    if weight_sum != 0:
        sent_vec /= weight_sum
    tfidf_sent_vectors.append(sent_vec)
    print(row, end=" ")
    row += 1
from numba import jit
def helper(sent, row):
    #global tfidf_sent_vectors
    sent_vec = np.zeros(50) # as word vectors are of zero length
    weight_sum =0; # num of words with a valid vector in the sentence/review
    for word in sent: # for each word in a review/sentence
        #if word in w2v_words:
        try:
            vec = w2v_model.wv[word]
            # obtain the tf_idfidf of a word in a sentence/review
            tf_idf = final_tf_idf[row, tfidf_feat.index(word)]
            sent_vec += (vec * tf_idf)
            weight_sum += tf_idf
        except:
            pass
    if weight_sum != 0:
        sent_vec /= weight_sum
    #tfidf_sent_vectors.append(sent_vec)
    #row += 1
    #print(row)
    return sent_vec, row
helper_numba = jit()(helper)
# TF-IDF weighted Word2Vec
tfidf_feat = tf_idf_vect.get_feature_names() # tfidf words/col-names
# final_tf_idf is the sparse matrix with row= sentence, col=word and cell_val = tfidf

tfidf_sent_vectors = [] # the tfidf-w2v for each sentence/review is stored in this list
#row=0;
for idx, sent in enumerate(list_of_sent): # for each review/sentence 
    vec, i = helper_numba(sent, idx)
    print(i, end=" ")
    tfidf_sent_vectors.append(vec)

# with ThreadPoolExecutor(max_workers=20) as executor:
#     result_futures = [executor.submit(helper_numba, sent=x, row=y) for y, x in enumerate(list_of_sent)]
#     for f in futures.as_completed(result_futures):
#         vec, i = f.result()
#         print(i, end=" ")
#         tfidf_sent_vectors.append(vec)
# print("Threading done!")

# for y, x in enumerate(list_of_sent):
#     i = helper_numba(sent=x, row=y)
#     print(i)
print("Done!")
#saving vectors into a pickle file
#for now, will be saving all vectors in one file
#will randomize and split everytime I load

#saving BoW unigrams
with open(os.path.join(dir_path, 'datasets','amzn_fine_food_reviews', 'bow_uni_vec.pkl'), 'wb') as bow:
    pickle.dump(final_counts, bow)

#saving BoW bigrams
with open(os.path.join(dir_path, 'datasets','amzn_fine_food_reviews', 'bow_bi_vec.pkl'), 'wb') as bi_bow:
    pickle.dump(final_bigram_counts, bi_bow)

#saving tfidf
with open(os.path.join(dir_path, 'datasets','amzn_fine_food_reviews', 'tfidf_vec.pkl'), 'wb') as tfidf_pickle:
    pickle.dump(final_tf_idf, tfidf_pickle)

#saving word2vec
with open(os.path.join(dir_path, 'datasets','amzn_fine_food_reviews', 'w2v_vec.pkl'), 'wb') as w2v_pickle:
    pickle.dump(sent_vectors, w2v_pickle)

#saving tfidf weighted w2v
with open(os.path.join(dir_path, 'datasets','amzn_fine_food_reviews', 'tfidf_weighted_w2v_vec.pkl'), 'wb') as tfidf_w2v_pickle:
    pickle.dump(tfidf_sent_vectors, tfidf_w2v_pickle)

print("Saving pickle files done!")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11087176887987836, 'fraction_numerical': 0.012607704004054738, 'mean_word_length': 4.376362397820164, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '57022', 'license': 'Apache-2.0', 'n_tokens_mistral': 5305, 'n_tokens_neox': 5071, 'n_words': 1509, 'path': '2.assign_amzn_fine_food_review_tsne/AmazonFineFoodReviewsPreProcessing.ipynb', 'repo_name': 'be-shekhar/learning-ml', 'size': '15784'}