<jupyter_start><jupyter_text>Objective:Our main objective for this analysis is to train a model which can seperate the postive and negative reviews.In this problem we will apply clustering technique called DBSCAN (Density-based spatial clustering of applications with Noise) to get an idea if the data can be clustered based on the reviews the model finds to be similar.  Importing Libraries & getting Data<jupyter_code>import numpy as np 
import pandas as pd 
import sqlite3 
import matplotlib.pyplot as plt 
import seaborn as sns 

import re 
import math 
import os 
import string 
from tqdm import tqdm 

import warnings 
warnings.filterwarnings('ignore')

# Metrics libraries
from sklearn.metrics import accuracy_score ,confusion_matrix, roc_curve, auc ,roc_auc_score

# NLP libraries 
import nltk 
from nltk.corpus import stopwords 
from nltk.stem import PorterStemmer 
from nltk.stem.wordnet import WordNetLemmatizer 
from sklearn.feature_extraction.text import CountVectorizer 
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.feature_extraction.text import TfidfTransformer

from gensim.models import Word2Vec 
from gensim.models import KeyedVectors

import tensorflow as tf
from tensorflow.python.client import device_lib
os.environ['TF_FORCE_GPU_ALLOW_GROWTH'] = 'true'
physical_devices = tf.config.experimental.list_physical_devices('GPU')
assert len(physical_devices) > 0, "Not enough GPU hardware devices available"
config = tf.config.experimental.set_memory_growth(physical_devices[0], True)

print("Num GPUs Available: ", len(
    tf.config.experimental.list_physical_devices('GPU')))
print(device_lib.list_local_devices())
connection = sqlite3.connect(
    'G:/My Drive/datasets/amazon food review/dataset/database.sqlite')

# ignoring datapoints where score = 3 (neutral review)

filtered_data = pd.read_sql_query("""
SELECT * 
FROM Reviews
WHERE Score != 3
LIMIT 50000
""", connection)

# label encoding score into positive (x>3) and negative (x<3)

def partition(x):
    if x < 3:
        return 0
    else:
        return 1


score_with_neutral = filtered_data['Score']
score_wo_neutral = score_with_neutral.map(partition)

filtered_data['Score'] = score_wo_neutral
filtered_data.head()
filtered_data.info()
filtered_data.shape
display = pd.read_sql_query("""
SELECT UserId, ProductId ,ProfileName ,Time, Score ,Text ,COUNT(*)
FROM Reviews 
GROUP BY UserId 
HAVING COUNT(*) > 1
""", connection)

display.head()
display.shape
display['COUNT(*)'].sum()<jupyter_output><empty_output><jupyter_text>Data Cleaning (Removing Duplicates)<jupyter_code>display = pd.read_sql_query("""
SELECT * 
FROM Reviews 
WHERE Score != 3 AND UserId='#oc-R11D9D7SHXIJB9'
ORDER BY ProductId
""", connection)

display.head()<jupyter_output><empty_output><jupyter_text>NOTE : - It can be seen that same user has multiple reviews with same values for all the parameters.- So, in order to get unbiased results ,we need to remove the rows having same parameters.- Steps to do so :     - first sort the data according to ProductId in ascending order.    - Then just keep the first similar product review and delete the others<jupyter_code>sorted_data = filtered_data.sort_values(
    'ProductId', axis=0, ascending=True, inplace=False, kind='quicksort', na_position='last')
clean_data = sorted_data.drop_duplicates(
    subset={"UserId", "ProfileName", "Time", "Text"}, keep='first', inplace=False)

clean_data.shape<jupyter_output><empty_output><jupyter_text>Data Preprocessing  Steps : 1. Begin by removing the HTML tags2. Remove any punctuations or limited set of special characters like , or . etc3. Check if the word is made up of english letters and is not alpha-numeric.4. Check to see if the length of the word is greater than 2 (as it was researched that there is no adjective in 2-letters)5. Convert the words to lowercase6. Remove Stopwords7. Stemming the word<jupyter_code># sample reviews

review_1 = clean_data['Text'].values[0]
print(review_1)
print('---------------------------------------')

review_2 = clean_data['Text'].values[1]
print(review_2)
print('---------------------------------------')

review_3 = clean_data['Text'].values[2]
print(review_3)<jupyter_output>My dogs loves this chicken but its a product from China, so we wont be buying it anymore.  Its very hard to find any chicken products made in the USA but they are out there, but this one isnt.  Its too bad too because its a good product but I wont take any chances till they know what is going on with the china imports.
---------------------------------------
Our dogs just love them.  I saw them in a pet store and a tag was attached regarding them being made in China and it satisfied me that they were safe.
---------------------------------------
Why is this $[...] when the same product is available for $[...] here?<br />http://www.amazon.com/VICTOR-FLY-MAGNET-BAIT-REFILL/dp/B00004RBDY<br /><br />The Victor M380 and M502 traps are unreal, of course -- total fly genocide. Pretty stinky, but only right nearby.<jupyter_text>1. Removing URLs<jupyter_code>review_1 = re.sub('http\S+', "", review_1)
review_2 = re.sub('http\S+', "", review_2)
review_3 = re.sub('http\S+', "", review_3)<jupyter_output><empty_output><jupyter_text>2. Removing HTML tags<jupyter_code>from bs4 import BeautifulSoup

soup = BeautifulSoup(review_1, 'lxml')
review_1 = soup.get_text()

soup = BeautifulSoup(review_2, 'lxml')
review_2 = soup.get_text()

soup = BeautifulSoup(review_3, 'lxml')
review_3 = soup.get_text()<jupyter_output><empty_output><jupyter_text>3. Removing Apostrophes<jupyter_code>def removing_apostrophes(text):
    # general
    text = re.sub("n\'t", " not", text)
    text = re.sub("\'re", " are", text)
    text = re.sub("\'s", " is", text)
    text = re.sub("\'d", " would", text)
    text = re.sub("\'ll", " will", text)
    text = re.sub("\'t", " not", text)
    text = re.sub("\'ve", " have", text)
    text = re.sub("\'m", " am", text)

    text = re.sub("won't", "will not", text)
    text = re.sub("can\'t", "can not", text)

    return text


# isn't gets converted to 'is not' in line 3
review_3 = removing_apostrophes(review_3)
print(review_3)<jupyter_output>Why is this $[...] when the same product is available for $[...] here? />The Victor M380 and M502 traps are unreal, of course -- total fly genocide. Pretty stinky, but only right nearby.<jupyter_text>4. Removing Numbers / Punctuations /Special Characters<jupyter_code># removing numbers
review_1 = re.sub('\S*\d\S*', "", review_1).strip()
print(review_1)
print('-----------------------------------------')

# removing special characters
review_3 = re.sub('[^A-Za-z0-9]+', ' ', review_3)
print(review_3)<jupyter_output>My dogs loves this chicken but its a product from China, so we wont be buying it anymore.  Its very hard to find any chicken products made in the USA but they are out there, but this one isnt.  Its too bad too because its a good product but I wont take any chances till they know what is going on with the china imports.
-----------------------------------------
Why is this when the same product is available for here The Victor M380 and M502 traps are unreal of course total fly genocide Pretty stinky but only right nearby<jupyter_text>5. Removing Stopwords<jupyter_code># removing the following words from the stop words list: 'no', 'nor', 'not'

stopwords = set(['br', 'the', 'i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're", "you've",
                "you'll", "you'd", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself',
                 'she', "she's", 'her', 'hers', 'herself', 'it', "it's", 'its', 'itself', 'they', 'them', 'their',
                 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', "that'll", 'these', 'those',
                 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does',
                 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of',
                 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after',
                 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further',
                 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more',
                 'most', 'other', 'some', 'such', 'only', 'own', 'same', 'so', 'than', 'too', 'very',
                 's', 't', 'can', 'will', 'just', 'don', "don't", 'should', "should've", 'now', 'd', 'll', 'm', 'o', 're',
                 've', 'y', 'ain', 'aren', "aren't", 'couldn', "couldn't", 'didn', "didn't", 'doesn', "doesn't", 'hadn',
                 "hadn't", 'hasn', "hasn't", 'haven', "haven't", 'isn', "isn't", 'ma', 'mightn', "mightn't", 'mustn',
                 "mustn't", 'needn', "needn't", 'shan', "shan't", 'shouldn', "shouldn't", 'wasn', "wasn't", 'weren', "weren't",
                 'won', "won't", 'wouldn', "wouldn't"])<jupyter_output><empty_output><jupyter_text>Combining all Steps<jupyter_code>from tqdm import tqdm
preprocessed_reviews = []

for text in tqdm(clean_data['Text'].values):
    text = re.sub('http\S+', "", text)
    text = BeautifulSoup(text, 'lxml').get_text()

    text = removing_apostrophes(text)

    text = re.sub('\S*\d\S*', "", text).strip()
    text = re.sub('[^A-Za-z0-9]+', ' ', text)

    text = ' '.join(i.lower()
                    for i in text.split() if i.lower() not in stopwords)

    preprocessed_reviews.append(text.strip())<jupyter_output>100%|██████████| 46072/46072 [00:24<00:00, 1863.07it/s]<jupyter_text>Splitting into train and test sets<jupyter_code>preprocessed_reviews = preprocessed_reviews[:10000]

X = preprocessed_reviews
y = np.array(clean_data['Score'])<jupyter_output><empty_output><jupyter_text>WordCloud<jupyter_code>def wordcloud(model, optimal_K, X):
    labels = model.labels_
    cluster_dict = {i: np.where(labels == i)[0] for i in range(optimal_K)}

    for cluster_number in range(optimal_K):
        cluster = [cluster_dict[cluster_number][i]
                   for i in range(cluster_dict[cluster_number].size)]

        reviews_cluster = []
        for i in cluster:
            reviews_cluster.append(X[i])

        review_corpus = ''
        for review in reviews_cluster:
            review_corpus = review_corpus + ' ' + review

        # wordcloud
        wordclouds = WordCloud(width=800, height=400, margin=2, scale=1, max_words=75, min_font_size=5, random_state=42,
                               background_color='black', contour_color='black', repeat=False).generate(str(review_corpus))

        plt.figure(figsize=(16, 8))
        plt.title("WordCloud for Cluster {}".format(cluster_number))
        plt.imshow(wordclouds, interpolation='bilinear')
        plt.axis('off')
        plt.show()<jupyter_output><empty_output><jupyter_text>Models<jupyter_code>from sklearn.preprocessing import StandardScaler

from sklearn.cluster import DBSCAN
from sklearn.neighbors import NearestNeighbors

from wordcloud import WordCloud<jupyter_output><empty_output><jupyter_text>1. AVG-W2V<jupyter_code>#word2vec for train data

sent_list_train = []
sent_vectors_train = []

for sentence in X:
    sent_list_train.append(sentence.split())

w2v_model = Word2Vec(sent_list_train, min_count=5, vector_size=50, workers=4)
w2v_words = list(w2v_model.wv.index_to_key)


for sent in tqdm(sent_list_train):
    sent_vectors = np.zeros(50)
    count_words = 0

    for word in sent:
        if word in w2v_words:
            vectors = w2v_model.wv[word]
            sent_vectors += vectors
            count_words += 1

    if count_words != 0:
        sent_vectors /= count_words

    sent_vectors_train.append(sent_vectors)

print(len(sent_vectors_train))
print(len(sent_vectors_train[0]))
scaler = StandardScaler()
data_avg = scaler.fit_transform(sent_vectors_train)

min_pts = 100<jupyter_output><empty_output><jupyter_text>1.1 Finding value of epsilon (AVG-W2V)<jupyter_code># calculating the distance to the nearest n-points for each point

distance = []
for i in data_avg:
    value = np.sort(np.sum((data_avg-i)**2 ,axis=-1))
    distance.append(value[min_pts])

final_epsilon = np.sqrt(np.array(distance))

# sorting 
sorted_distance = np.sort(final_epsilon)
points = [point for point in range(len(sent_vectors_train))]
plt.figure(figsize=(10,5))
plt.plot(points, sorted_distance)
plt.title('K-Distance Graph AVG-W2V')

plt.xlabel('Data-Points sorted by Distance')
plt.ylabel('eps')

plt.grid()
plt.show()<jupyter_output><empty_output><jupyter_text>The point of maximum curvature in the above given K-distance graph is : 8<jupyter_code>optimal_eps_avg_w2v = 8<jupyter_output><empty_output><jupyter_text>1.2 Model-Building (AVG-W2V)<jupyter_code>model_avg_w2v = DBSCAN(eps=optimal_eps_avg_w2v)
model_avg_w2v.fit(data_avg)

y_pred_avg_w2v = model_avg_w2v.fit_predict(data_avg)<jupyter_output><empty_output><jupyter_text>1.3 Cluster Analysis (AVG-W2V)<jupyter_code>cluster_num = pd.DataFrame(y_pred_avg_w2v, columns=['Cluster_number'])
cluster_num
new_data = pd.concat([clean_data, cluster_num], axis=1)
new_data.head()
new_data.groupby(['Cluster_number'])['Text'].count()
plt.bar([x for x in range(3)], new_data.groupby(['Cluster_number'])['Text'].count(), alpha=0.5)

plt.title('Number of Reviews per Cluster')
plt.grid()
plt.xlabel('Cluster Number')
plt.ylabel('Number of Reviews')
plt.show()<jupyter_output><empty_output><jupyter_text>1.4 WordCloud (AVG-W2V)<jupyter_code>wordcloud(model_avg_w2v ,1 ,X)<jupyter_output><empty_output><jupyter_text>2. TFIDF-W2V<jupyter_code>sent_list_train = []

for sentence in X:
    sent_list_train.append(sentence.split())
    
w2v_model = Word2Vec(sent_list_train ,min_count=5 ,vector_size=50 ,workers=4)
w2v_words = list(w2v_model.wv.index_to_key)


tfidf_vectors = TfidfVectorizer(ngram_range=(1,2) ,min_df=10 ,max_features=500)
tfidf_matrix = tfidf_vectors.fit_transform(X)

tfidf_features = tfidf_vectors.get_feature_names()
tfidf_dictionary = dict(zip(tfidf_vectors.get_feature_names() ,list(tfidf_vectors.idf_)))


#word2vec for train data

tfidf_sent_vectors_train = []
row = 0

for sent in tqdm(sent_list_train):
    sent_vectors = np.zeros(50)
    weight_sum = 0 
    
    for word in sent:
        if word in w2v_words and word in tfidf_features:
            vectors = w2v_model.wv[word]
            tfidf = tfidf_dictionary[word]*(sent.count(word) / len(sent))
            sent_vectors += (vectors * tfidf) 
            weight_sum += tfidf
            
    if weight_sum != 0:
        sent_vectors /= weight_sum 
        
    tfidf_sent_vectors_train.append(sent_vectors)
    row +=1
scaler = StandardScaler()
data_tfidf = scaler.fit_transform(tfidf_sent_vectors_train)

min_pts = 100<jupyter_output><empty_output><jupyter_text>2.1 Finding value of epsilon (TFIDF-W2V)<jupyter_code># calculating the distance to the nearest n points for each point

distance = []
for i in data_tfidf:
    value = np.sort(np.sum((data_tfidf-i)**2, axis=-1))
    distance.append(value[min_pts])

final_epsilon = np.sqrt(np.array(distance))

# sorting
sorted_distance = np.sort(final_epsilon)
points = [point for point in range(len(tfidf_sent_vectors_train))]

plt.figure(figsize=(10, 5))
plt.plot(points, sorted_distance)
plt.title('K-Distance Graph TFIDF-W2V')

plt.xlabel('Data Points sorted by Distance')
plt.ylabel('eps')

plt.grid()
plt.show()<jupyter_output><empty_output><jupyter_text>The point of maximum curvature in the above given K-distance graph is : 7<jupyter_code>optimal_eps_tfidf_w2v = 7<jupyter_output><empty_output><jupyter_text>2.2 Model-Building (TFIDF-W2V)<jupyter_code>model_tfidf_w2v = DBSCAN(eps=optimal_eps_tfidf_w2v)
model_tfidf_w2v.fit(data_tfidf)

y_pred_tfidf_w2v = model_tfidf_w2v.fit_predict(data_tfidf)<jupyter_output><empty_output><jupyter_text>2.3 Cluster Analysis (TFIDF-W2V)<jupyter_code>cluster_num = pd.DataFrame(y_pred_tfidf_w2v, columns=['Cluster_number'])
cluster_num
new_data = pd.concat([clean_data, cluster_num], axis=1)
new_data.head()
new_data.groupby(['Cluster_number'])['Text'].count()
plt.bar([x for x in range(3)], new_data.groupby(['Cluster_number'])['Text'].count(), alpha=0.5)

plt.title('Number of Reviews per Cluster')
plt.grid()
plt.xlabel('Cluster Number')
plt.ylabel('Number of Reviews')
plt.show()<jupyter_output><empty_output><jupyter_text>2.4 WordCloud (TFIDF-W2V)<jupyter_code>wordcloud(model_tfidf_w2v ,1 ,X)<jupyter_output><empty_output><jupyter_text>Model Comparision  1. Silhoutte Score<jupyter_code>from sklearn.metrics import silhouette_score

avg_w2v_score = silhouette_score(data_avg, y_pred_avg_w2v)
print('Silhoutte Score for AVG-W2V is : {}'.format(avg_w2v_score))

tfidf_w2v_score = silhouette_score(data_tfidf, y_pred_tfidf_w2v)
print('Silhoutte Score for TFIDF-W2V is : {}'.format(tfidf_w2v_score))<jupyter_output>Silhoutte Score for AVG-W2V is : 0.4228260860985999
Silhoutte Score for TFIDF-W2V is : 0.28778599498728535<jupyter_text>2. Creating a prettytable to display comparision<jupyter_code>from prettytable import PrettyTable
x = PrettyTable()
x.title = 'DBSCAN'

x.field_names = ['Vectorizer', 'Optimal-Clusters', 'Silhoutte-Score']
x.add_row(['AVG-W2V', 4, 0.4228])
x.add_row(['TFIDF-W2V', 3, 0.2877])

x.sortby = 'Silhoutte-Score'
x.reversesort = True
print(x)<jupyter_output>+-------------------------------------------------+
|                      DBSCAN                     |
+------------+------------------+-----------------+
| Vectorizer | Optimal-Clusters | Silhoutte-Score |
+------------+------------------+-----------------+
|  AVG-W2V   |        4         |      0.4228     |
| TFIDF-W2V  |        3         |      0.2877     |
+------------+------------------+-----------------+
{'fraction_non_alphanumeric': 0.1574396406513195, 'fraction_numerical': 0.019034250421111735, 'mean_word_length': 4.12546762589928, 'pattern_counts': {'":': 0, '<': 109, '<?xml version=': 0, '>': 110, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '337622', 'license': 'MIT', 'n_tokens_mistral': 6497, 'n_tokens_neox': 6096, 'n_words': 1501, 'path': 'Clustering Models/DBSCAN/amazon-dbscan.ipynb', 'repo_name': 'faizanxmulla/amazon-fine-food-reviews', 'size': '17810'}