<jupyter_start><jupyter_text>Sentiment Analysis in KerasIn this notebook we want to train a neural network in Keras in order to predict the sentiment of a movie review, i.e. whether it is positive or negative. For the purpose, the imdb movie review dataset available in Keras is used. This problem has been widely addressed and is one of the example use cases of Keras, useful to experiment and learn.- We rely on the examples provided by the Coursera course https://www.coursera.org/learn/ai/lecture/hQYsN/recurrent-neural-networks, https://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/ and http://www.samyzaf.com/ML/imdb/imdb.html- Embeddings are used to represent words and provide them as input to the neural network: https://en.wikipedia.org/wiki/Embedding, https://en.wikipedia.org/wiki/Word_embedding, https://keras.io/layers/embeddings/- Convolutional neural nets were shown leading good results in spite of small network structure: http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/, https://github.com/alexander-rakhlin/CNN-for-Sentence-Classification-in-Keras/blob/master/docs/1408.5882v2.pdf, http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/- A more complete overview at http://cs224d.stanford.edu/syllabus.html<jupyter_code>from keras.datasets import imdb

# select max_features most common items from vocabulary
max_features = 20000
top_words = max_features
max_review_length = 500
(x_train, y_train), (x_test, y_test) = imdb.load_data(path="imdb.npz",
                                                      num_words=top_words,
                                                      skip_top=0,
                                                      maxlen=max_review_length,
                                                      seed=113,
                                                      start_char=1,
                                                      oov_char=2,
                                                      index_from=3)
print x_train.shape
print y_train.shape
print x_test.shape
print y_test.shape<jupyter_output>(25000,)
(25000,)
(20947,)
(20947,)<jupyter_text>Since the reviews might have different lenght and we have a fixed input for our network to train, we use padding to create sub-sequences of fixed length, i.e. if the raw sequence is longer than the set size it will truncate it, while if shorter than the set size the padding will add zeros to fill the space.<jupyter_code>from keras.preprocessing import sequence

x_train = sequence.pad_sequences(x_train, maxlen=max_review_length)
print x_train.shape
x_test = sequence.pad_sequences(x_test, maxlen=max_review_length)
print x_test.shape<jupyter_output>(25000, 500)
(20947, 500)<jupyter_text>For the model we use:1. An input Embedding layer using 128-length vectors to represent each word. 2. An LSTM layer with 128 memory units3. A Dense output layer with a single neuron, whose sigmoid activation will define the belonging class for the review<jupyter_code>from keras.models import Sequential
from keras.layers import Embedding, LSTM, Dense
#from keras.layers.embeddings import Embedding

embedding_vector_length = 128

model = Sequential()
# input layer
model.add(Embedding(max_features,128
                    #input_dim=top_words,
                    #output_dim=embedding_vector_length,
                    #input_length=max_review_length
                   ))
# hidden layer
model.add(LSTM(units=128
               #units=128
               #dropout=0.2,
               #recurrent_dropout=0.2
              ))

# output layer
model.add(Dense(1,
                activation='sigmoid'))

model.compile(loss='binary_crossentropy',
              #optimizer='adam',
              optimizer='sgd',
              metrics=['accuracy'])

print(model.summary())<jupyter_output>_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_1 (Embedding)      (None, None, 128)         2560000   
_________________________________________________________________
lstm_1 (LSTM)                (None, 128)               131584    
_________________________________________________________________
dense_1 (Dense)              (None, 1)                 129       
=================================================================
Total params: 2,691,713
Trainable params: 2,691,713
Non-trainable params: 0
_________________________________________________________________
None<jupyter_text>Set the seed and train the model:<jupyter_code># fix random seed for reproducibility
import numpy
numpy.random.seed(7)

# define the batch size, ie. the size of the set (or number of observations) used between weight updates
# this is a tradeoff as bigger batch sizes allow for learning bigger dependencies across data samples
# although brings in greater computational complexity
# viceversa smaller batch sizer are lighter to handle and might lead to better weight updates for non-sequential data
batch_size = 32

model.fit(x_train,
          y_train,
          validation_data=(x_test, y_test),
          epochs=1,
          batch_size=batch_size)<jupyter_output>Train on 25000 samples, validate on 20947 samples
Epoch 1/1
25000/25000 [==============================] - 438s 18ms/step - loss: 0.6928 - acc: 0.5107 - val_loss: 0.6928 - val_acc: 0.5336<jupyter_text>Evaluate the model:<jupyter_code>loss, accuracy = model.evaluate(x_train, y_train, batch_size=batch_size)
print("Training: accuracy = %f  ;  loss = %f" % (accuracy, loss))
loss, accuracy = model.evaluate(x_test, y_test, batch_size=batch_size)
print("Testing: accuracy = %f  ;  loss = %f" % (accuracy, loss))<jupyter_output>20947/20947 [==============================] - 67s 3ms/step<jupyter_text>Save the model:<jupyter_code>model.save("imdb.h5")
!ls<jupyter_output>imdb.h5       imdb_sa.ipynb<jupyter_text>which can be reloaded as:<jupyter_code>from keras.models import load_model
model = load_model("imdb.h5")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10908795317834498, 'fraction_numerical': 0.03397821492440253, 'mean_word_length': 3.248618784530387, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 6, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '509266', 'license': 'Unlicense', 'n_tokens_mistral': 1878, 'n_tokens_neox': 1687, 'n_words': 484, 'path': 'imdb_sentiment/imdb_sa.ipynb', 'repo_name': 'pilillo/nlp-notebooks', 'size': '6151'}