<jupyter_start><jupyter_text>IMPLEMENTATION OF DEEP LEARNING MODELS FOR SENTIMENT ANALYSIS OF AMAZON REVIEWS.* This notebooks contains the implementation of a Deep Neural Network (DNN), Convolutional Neural Network (CNN), and recurrent Neural Network (LSTM) for sentiment analysis.* The dataset used is **"Sentiment Labelled Sentences Dataset", from the UC Irvine Machine Learning Repository.*** The sentences come from three different websites/fields:    * amazon.com    * imdb.com    * yelp.com* In this notebook, only the Amazon product reviews dataset is used.* Each sentence is labelled as either 1 (for positive) or 0 (for negative).* There exist 500 positive and 500 negative sentences.* This dataset was created for the Paper 'From Group to Individual Labels using Deep Features', Kotzias et. al,. KDD 2015.  *(Please cite the paper if you want to use it :))** Link to the dataset is: [Sentiment Labelled Sentences Data Set](https://archive.ics.uci.edu/ml/datasets/Sentiment+Labelled+Sentences)  Importing modules:We are importing two custom built libraries namely "TextPreprocessing.py" and "Model.py".1. The **TextPreprocessing.py** file contains methods for reading the dataset, preprocessing it, generating tokens, and word embeddings.2. The **Model.py** file contains callback class as well as methods in which the DNN, CNN and RNN architectures are defined. It also contains methods to save, and plot models as well as predict class label.<jupyter_code>from TextPreprocessing import TextPreprocessing
import Model as model

from sklearn.model_selection import train_test_split
import tensorflow as tf
from tensorflow import keras
from keras.regularizers import l2
import numpy as np
import pandas as pd
max_words = 10000
oov_word = "<00V>"
padding_type = "post"
truncating_type = "post"
pad_len = 100<jupyter_output><empty_output><jupyter_text>* Create an object of TextPreprocessing class.* This calls contains all the methods for text Preprocesing and vectorizaiton.* Passing "Amazon" as the argument to the consturctor indicates that it must use the Amazon product reviews dataset.<jupyter_code>tp = TextPreprocessing('Amazon')
corpus, labels = tp.get_data()
df = pd.DataFrame(list(zip(corpus,labels)), columns=['corpus', 'labels'])
df.head()
# Preprocess the dataset
X = []
for c in df['corpus'].values:
    X.append(tp.preprocess_text(c))
print(df['corpus'].values[131])
print(X[131])
print(df['labels'].values[131])

print("\n"+df['corpus'].values[43])
print(X[43])
print(df['labels'].values[43])
#Splitting into training(80%) and testing data(20%)
X_train, X_test, y_train, y_test = train_test_split(X, df['labels'].values, test_size=0.20, random_state=42, shuffle=True)
y_train = y_train.reshape(y_train.shape[0], -1)
y_train = np.array(y_train, dtype=np.float32)
y_test = y_test.reshape(y_test.shape[0], -1)
y_test = np.array(y_test, dtype=np.float32)
#Tokenize and Pad the text sequences
tokenizer, vocab_size, X_train_padded, X_test_padded = tp.tokenizer_and_pad_training(X_train, X_test, max_words, oov_word, padding_type, truncating_type, pad_len)
print(X_train_padded.shape)
print(y_train.shape)
print(X_test_padded.shape)
print(y_test.shape)<jupyter_output>(800, 100)
(800, 1)
(200, 100)
(200, 1)<jupyter_text>Word Embeddings:* GloVe Word Embeddings are used. * Each word is represented as a 100 dimensional feature vector.* The file contains embedings for the most frequently used 6 billion words in the english language.<jupyter_code>embedding_matrix = tp.get_embedding_metrix(vocab_size, tokenizer)
print(embedding_matrix.shape)
input_dim = vocab_size
embedding_dim = 100
input_len = pad_len

myCB = model.myCallbacks()<jupyter_output><empty_output><jupyter_text>Deep Neural Network Model:* A DNN architecture consists of an Embedding layer, followed by a GlobalMaxPooling1D and three Dense layers.* Pretained GloVe Word Embeddings are used which are again trained according to the dataset.* First Dense layer contains 64 hidden units, and second Dense layer contains 32 hidden units with relu activation.* Weight decay / L2 regularization with lambda=0.015 is used in both the dense layers to reduce overfitting.* The output layer is a Dense layer with a single hidden unit using softmax activation.* Adam optimizer and binary crossentropy loss is used.* The model is trained with a batch size of 32 for 40 epochs / unitl training accuracy reaches 99%.<jupyter_code>myDNN = model.create_model_DNN(input_dim, embedding_dim, embedding_matrix, input_len, trainable=True, n1=64, n2=32, kr=l2(0.015))
myDNN.summary()
history = myDNN.fit(X_train_padded, y_train, epochs=40, batch_size=32, verbose=1, validation_data = (X_test_padded, y_test), callbacks=[myCB])
model.plot_curves(history)<jupyter_output><empty_output><jupyter_text>Results:* We get a training accuracy of 99.00% and testing accuracy of 81.50%.* The training loss is 0.2560 and testing loss is 0.5575.<jupyter_code>model.save_models(myDNN, "DNN_n164_n232_kr0015_vacc81")<jupyter_output><empty_output><jupyter_text>Convolutional Neural Network Model:* A CNN architecture consists of an Embedding layer, followed by a Conv21D layer, GlobalMaxPooling1D layer, Dropout Layer and Dense layer.* Pretained GloVe Word Embeddings are used which are again trained according to the dataset.* The Conv21D layer contains 64 hidden units, with kernel size 5.* Weight decay / L2 regularization with lambda=0.02 is used for the Convolutional layer to reduce overfitting.* A dropout of 30% is implemented.* The output layer is a Dense layer with a single hidden unit using softmax activation.* Adam optimizer and binary crossentropy loss is used.* The model is trained with a batch size of 32 for 40 epochs / unitl training accuracy reaches 99%.<jupyter_code>myCNN = model.create_model_CNN(input_dim, embedding_dim, embedding_matrix, input_len, trainable=True, n1=64, k=5, d=0.3, kr=l2(0.02))
myCNN.summary()
history1 = myCNN.fit(X_train_padded, y_train, epochs=40, batch_size=32, validation_data=(X_test_padded, y_test), callbacks=[myCB], verbose=1)
myCNN.evaluate(X_test_padded, y_test, verbose=1)
model.plot_curves(history1)<jupyter_output><empty_output><jupyter_text>Results:* We get a training accuracy of 99.00% and testing accuracy of 87.00%.* The training loss is 0.1766 and testing loss is 0.4033.<jupyter_code>model.save_models(myCNN, "CNN_n164_k5_d03_kr002_vacc87")<jupyter_output><empty_output><jupyter_text>Recurrent Neural Network Model:* A RNN architecture consists of an Embedding layer, followed by a Bidirectional LSTM layer, GlobalMaxPooling1D layer, and two Dense layers.* Pretained GloVe Word Embeddings are used which are again trained according to the dataset.* The Bidirectional LSTM layer contains 64 hidden units.* A dropout of 30% is implemented.* The following Dense layer contains 32 hidden units with relu activation.* The output layer is a Dense layer with a single hidden unit using softmax activation.* Adam optimizer and binary crossentropy loss is used.* The model is trained with a batch size of 32 for 40 epochs / unitl training accuracy reaches 99%.<jupyter_code>myLSTM = model.create_model_LSTM(input_dim, embedding_dim, embedding_matrix, input_len, trainable=True, n1=64, n2=32, d=0.3)
myLSTM.summary()
history2 = myLSTM.fit(X_train_padded, y_train, epochs=50, batch_size=32, validation_data=(X_test_padded, y_test), callbacks=[myCB], verbose=1)
myLSTM.evaluate(X_test_padded, y_test)
model.plot_curves(history2)<jupyter_output><empty_output><jupyter_text>Results:* We get a training accuracy of 99.00% and testing accuracy of 87.00%.* The training loss is 0.0279 and testing loss is 0.6260 .<jupyter_code>model.save_models(myLSTM, "LSTM_n164_n232_d03_vacc87")<jupyter_output><empty_output><jupyter_text>Testing on randomly collected samples<jupyter_code>DNN = tf.keras.models.load_model("../models/DL/Amazon/DNN_n164_n232_kr0015_vacc81.h5")
CNN = tf.keras.models.load_model("../models/DL/Amazon/CNN_n164_k5_d03_kr002_vacc87.h5")
RNN = tf.keras.models.load_model("../models/DL/Amazon/LSTM_n164_n232_d03_vacc87.h5")
test_case = ["Phone is defective, volume botton and power button not working properly. What can i do please suggest.", 
             "Myntra is a very BAD app, very expensive as compared to others. Not affordable for common man. Very low quality.",
            "I really love shopping at Amazon because any time I have ever had a problem their customer service has always been great about fixing the issue. ",
            "Delivery is very quick also even at the busiest times of the year. I find my orders can come quicker than the estimated date.",
            "I purchased a item from Amazon and my debit card was charge THREE times! Customer service was terrible! No refund or credit was issued as of now! Amazon has terrible business practices!",
            "I absolutely love Amazon. Their selections, prices, notifications and returns are so easy and customer friendly."]
print(len(test_case))
test_input = []
for test in test_case:
    instance = tp.preprocess_and_tokenize_test_case(tokenizer, test, padding_type, truncating_type, pad_len)
    test_input.append(instance)
print(len(test_input))
for x in test_input:
    label, confidence = model.predict_label(DNN.predict(x))
    print("label is :{}, and it's confidence is: {}".format(label, confidence))
for x in test_input:
    label, confidence = model.predict_label(CNN.predict(x))
    print("label is :{}, and it's confidence is: {}".format(label, confidence))
for x in test_input:
    label, confidence = model.predict_label(RNN.predict(x))
    print("label is :{}, and it's confidence is: {}".format(label, confidence))<jupyter_output>label is :negative, and it's confidence is: 0.00018255710892844945
label is :negative, and it's confidence is: 8.313765283674002e-05
label is :positive, and it's confidence is: 0.9999982118606567
label is :positive, and it's confidence is: 0.9999985694885254
label is :negative, and it's confidence is: 1.3961129297968e-05
label is :positive, and it's confidence is: 0.9999910593032837
{'fraction_non_alphanumeric': 0.0890397284887203, 'fraction_numerical': 0.04102615292473548, 'mean_word_length': 5.857631759069131, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 40, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '335370', 'license': 'MIT', 'n_tokens_mistral': 3394, 'n_tokens_neox': 3047, 'n_words': 1098, 'path': 'modules/Amazon_Sentiment_Analysis_DL.ipynb', 'repo_name': 'mr-atharva-kulkarni/UCI-Sentiment-Analysis', 'size': '10018'}