<jupyter_start><jupyter_text>Sentiment AnalysisWe will use the IMDB sentiment analysis database in this tutorial. The main idea that is used in this tutorial is that certain words are enough to establish the sentiment of a given sentence. The word order is discarded in this particular tutorial. References1. http://ai.stanford.edu/~amaas/data/sentiment/<jupyter_code>import numpy as np
import pandas as pd
import os
import urllib

from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer

import matplotlib.pyplot as plt
%matplotlib inline

np.random.seed(1)
if not os.path.isfile('data/reviews2.pkl'):
    urllib.request.urlretrieve('https://www.dropbox.com/s/15tfttuzqe7fimg/reviews2.pkl?dl=1','./data/reviews2.pkl')
    
df = pd.read_pickle('data/reviews2.pkl')    
df.head()
df.Reviews.values[0]
df.shape<jupyter_output><empty_output><jupyter_text>In the first model that we look at we simply count the number of words in each document. This method is called **Bag of words**. It's considered a bag since we lose all ordering of the words. The features are simply the count of each word. Lucky for us Scikit Learn contains this feature extractor as a method. See http://scikit-learn.org/stable/modules/feature_extraction.htmlcommon-vectorizer-usage for an example. the `fit_transform` method gives us the matrix that we are after.<jupyter_code>tf_vectorizer = CountVectorizer()
tf = tf_vectorizer.fit_transform(df.Reviews.values)
print('Number of unique words: ',len(tf_vectorizer.get_feature_names()))
print(tf_vectorizer.get_feature_names()[:5])<jupyter_output>Number of unique words:  74849
['00', '000', '0000000000001', '00001', '00015']<jupyter_text>Unfortunately this is extremely noisy as can be seen by the existence of random numbers such as `00` etc. Hence we will do two additional things. We will only consider words that have appeared atleast 5 times. Also words such as `the, a, to` are irrelevant in figuring out what the sentiment is. Hence we shall remove such words. These are commonly known as _stop words_ in Natural Language Processing (NLP).<jupyter_code>tf_vectorizer = CountVectorizer(min_df=5,stop_words='english')
tf = tf_vectorizer.fit_transform(df.Reviews.values)
print('New number of unique words: ',len(tf_vectorizer.get_feature_names()))<jupyter_output>New number of unique words:  26967<jupyter_text>Notice that tf is a sparse matrix<jupyter_code>print(type(tf))
tf.shape<jupyter_output><class 'scipy.sparse.csr.csr_matrix'><jupyter_text>We will use `np.random.permutation` to shuffle the data. It is really important to shuffle the data before we put it into any machine learning algorithm. See below as to how `np.random.permutation` works.<jupyter_code>np.random.permutation(5)
# TODO
idx = #fill out what is supposed to go in here (what is the length of all data)
X_train = tf[idx][:12500].todense()
X_test = tf[idx][12500:].todense()
y_train = df.Sentiment.values[idx][:12500]
y_test = df.Sentiment.values[idx][12500:]
X_train.shape<jupyter_output><empty_output><jupyter_text>Keras ModelBelow we create our first deep learning model.<jupyter_code>from keras.models import Sequential
from keras.layers import Dense, Activation, Dropout
from keras.regularizers import l2, l1
model = Sequential()
model.add(Dense(units=100, input_dim=tf.shape[1]))
model.add(Activation("relu"))
model.add(Dense(units=1))
model.add(Activation("sigmoid"))
model.compile(loss='binary_crossentropy', optimizer='adagrad', metrics=["binary_accuracy"])
model.summary()
model.fit(X_train, y_train, epochs=2, batch_size=128)<jupyter_output><empty_output><jupyter_text>Keras has two ways of predicting `model.predict(...)` which predicts the **probability** of belonging to a class (basically the value it gets after passing through final activation), and `model.predict_class(...)` which as the name suggests outputs the class. Here we will use the `predict` method.**This is the probability of the class being 1 (positive review in this case).**<jupyter_code>y_test_pred = model.predict(X_test)
print(y_test_pred.shape)
y_test_pred<jupyter_output>(12500, 1)<jupyter_text>Let us manually convert this to 0 or 1 class. Using numpy arrays<jupyter_code># TODO
# a probability of less than 0.5 is considered class 0
# Overwrite y_test to get the 0 classes
y_test_pred[y_test_pred>=0.5] = 1
np.count_nonzero(y_test_pred==y_test[:,None])*1.0/len(y_test)<jupyter_output><empty_output><jupyter_text>Let us test some manual test cases. Fee free to change the sentence below. Really important to note that I am using `transform` and not `fit_transform` below.<jupyter_code>test_case = tf_vectorizer.transform(["I really hated this movie"])
model.predict(test_case.todense())
test_case = tf_vectorizer.transform(["I really loved this movie"])
model.predict(test_case.todense())<jupyter_output><empty_output><jupyter_text>Improving the modelCounting is one of the most basic things you could do when it comes to NLP applications. A more popular method that tends to be a bit better is Term Frequency - inverse Document Frequency (Tfidf) method. Not only does it take frequncy (term frequency) into account, but it also penalises words for being overused (for example the word 'the' would be downvoted since it is too popular). The intuition being that it does not have any discriminatory power if the word is used in every sentence. Whereas rare words might be able to distinguish the contents of the sentence better.<jupyter_code>tfidf_vectorizer = TfidfVectorizer(min_df=5,stop_words='english')

# TODO
tfidf = #get the tfidf feature matrix (same as above)

# get the new features
X_train = tfidf[idx][:12500].todense()
X_test = tfidf[idx][12500:].todense()
model = Sequential()
# TODO:
# Complete the model, same/ similar as above.
# 1. Have two `Dense` layers
# 2. Compile the model (what is the loss in this case)
model.summary()
model.fit(X_train, y_train, epochs=2, batch_size=128)
y_test_pred = model.predict(X_test)
y_test_pred[y_test_pred<0.5] = 0
y_test_pred[y_test_pred>=0.5] = 1
np.count_nonzero(y_test_pred==y_test[:,None])/len(y_test)<jupyter_output><empty_output><jupyter_text>As can be seen the model is much better with our test data as seen below<jupyter_code>test_case = tf_vectorizer.transform(["I really hated this movie"])
model.predict(test_case.todense())
test_case = tf_vectorizer.transform(["I really loved this movie"])
model.predict(test_case.todense())
plt.hist(model.get_weights()[0].ravel(),100)
plt.show()
model.weights<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09978474784747847, 'fraction_numerical': 0.020141451414514146, 'mean_word_length': 6.268156424581005, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 44, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '86522', 'license': 'Apache-2.0', 'n_tokens_mistral': 2174, 'n_tokens_neox': 2072, 'n_words': 676, 'path': 'DL-Keras_Tensorflow/Lesson 05 - Keras Sentiment Analysis.ipynb', 'repo_name': 'eagleanurag/deepschool.io', 'size': '6504'}