<jupyter_start><jupyter_text>Clasificación de Sentimiento de Reviews de peliculas  INF-396 Introducción a la Ciencia de Datos  Autor: <NAME> Lenguaje: PythonTemas:    - Sentiment Analysis    - Feed Forward, RNN, LSTM    - Embedding GLOVE<jupyter_code>## Puede encontrar información sobre keras en https://keras.io/getting_started/intro_to_keras_for_engineers/
## Puede encontrar información sobre sklearn en https://scikit-learn.org/stable/

import os
import numpy as np
import pickle<jupyter_output><empty_output><jupyter_text>Primero, carguemos en memoria el conjunto de datos y veamos algunos de sus datos<jupyter_code>classes = ['negative', 'positive']
X_train, y_train, X_test, y_test = pickle.load(open('movies_review_train_test.dat', 'rb'))<jupyter_output><empty_output><jupyter_text>Modelos de redes neuronales<jupyter_code>from tensorflow import keras
from tensorflow.keras.models import Sequential, Model
from tensorflow.keras.layers import Dense, Input, Embedding, LSTM, SimpleRNN, Add, Lambda, Dropout
from tensorflow.keras.optimizers import Adam
import tensorflow.keras.backend as K
# Definición de parámetros
batch_size = 32
epochs = 2
num_classes = 2
max_words = 70
embedding_size = 30<jupyter_output><empty_output><jupyter_text>PreprocesamientoA continuación preprocesaremos los datos, realizando tokenización.<jupyter_code>from tensorflow.keras.preprocessing.text import Tokenizer

# Tokenizing text
tokenizer = Tokenizer()
tokenizer.fit_on_texts(X_train + X_test)
word_index = tokenizer.word_index
num_words = len(tokenizer.word_index) + 1
# Transformando texto a secuencias de indices
X_tr = tokenizer.texts_to_sequences(X_train)
X_te = tokenizer.texts_to_sequences(X_test)
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.utils import to_categorical
# Se agregan 0 a textos que tienen menos de max_words palabras y se cortan los que tienen mas
X_tr_pad = pad_sequences(X_tr, maxlen=max_words)
X_te_pad = pad_sequences(X_te, maxlen=max_words)
# Se convierten label a version categorica
y_train = to_categorical(y_train, num_classes=num_classes)
y_test = to_categorical(y_test, num_classes=num_classes)<jupyter_output><empty_output><jupyter_text>Entrenando los modelos  Red neuronal feedforwardPrimero entrenaremos una red neuronal feedforward directamente en el texto tokenizado, utilizando un embedding simple.<jupyter_code>document_input = Input(shape=(max_words, ), dtype='int32')


x = Embedding(num_words, embedding_size, input_length=max_words)(document_input)
f = Lambda(lambda x: K.sum(x,axis=1))(x)
y = Dense(20, activation='relu')(f)
output = Dense(num_classes, activation='softmax')(y)


# Construyendo el modelo
model = Model(inputs=[document_input], outputs=[output])
model.compile(loss='categorical_crossentropy', optimizer='adam',
             metrics=['accuracy'])

# Entrenando el modelo
history = model.fit(X_tr_pad, y_train,
                  batch_size=batch_size,
                  epochs=2,
                  verbose=1,
                  validation_split=0.2)
loss, acc = model.evaluate(X_te_pad, y_test)
print('Accuracy en datos de test: {0}'.format(acc))<jupyter_output>3000/3000 [==============================] - 0s 124us/sample - loss: 0.4422 - accuracy: 0.8187
Accuracy en datos de test: 0.8186666369438171<jupyter_text>Red Neuronal Recurrente Simple con Embedding<jupyter_code>document_input = Input(shape=(max_words, ), dtype='int32')
# Embedding Layer
embedding = Embedding(input_dim=num_words, output_dim=embedding_size, 
                      input_length=max_words, 
                      trainable=True)(document_input)

rnn = SimpleRNN(10,return_sequences=False)(embedding)



# Aplicando una capa densa luego de la RNN
output = Dense(num_classes, activation='softmax')(rnn)

# Construyendo el modelo
model = Model(inputs=[document_input], outputs=[output])
model.compile(loss='categorical_crossentropy', optimizer='adam',
             metrics=['accuracy'])

# Entrenando el modelo
history = model.fit(X_tr_pad, y_train,
                  batch_size=batch_size,
                  epochs=2,
                  verbose=1,
                  validation_split=0.2)
loss, acc = model.evaluate(X_te_pad, y_test)
print('Accuracy en datos de test: {0}'.format(acc))<jupyter_output>3000/3000 [==============================] - 1s 425us/sample - loss: 0.4975 - accuracy: 0.7603
Accuracy en datos de test: 0.7603333592414856<jupyter_text>Dropout para evitar el sobre ajuste<jupyter_code>document_input = Input(shape=(max_words, ), dtype='int32')
# Embedding Layer
embedding = Embedding(input_dim=num_words, output_dim=embedding_size, 
                      input_length=max_words, 
                      trainable=True)(document_input)


rnn = SimpleRNN(10,dropout=0.4,return_sequences=False)(embedding)

'''  -----------  '''

# Aplicando una capa densa luego de la RNN
output = Dense(num_classes, activation='softmax')(rnn)

# Construyendo el modelo
model = Model(inputs=[document_input], outputs=[output])
model.compile(loss='categorical_crossentropy', optimizer='adam',
             metrics=['accuracy'])

# Entrenando el modelo
history = model.fit(X_tr_pad, y_train,
                  batch_size=batch_size,
                  epochs=2,
                  verbose=1,
                  validation_split=0.2)<jupyter_output>WARNING:tensorflow:From /home/leicasdayz/anaconda3/lib/python3.7/site-packages/tensorflow/python/keras/backend.py:4081: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
Train on 12000 samples, validate on 3000 samples
Epoch 1/2
12000/12000 [==============================] - 39s 3ms/sample - loss: 0.6243 - accuracy: 0.6433 - val_loss: 0.4818 - val_accuracy: 0.7687
Epoch 2/2
12000/12000 [==============================] - 38s 3ms/sample - loss: 0.3619 - accuracy: 0.8530 - val_loss: 0.4005 - val_accuracy: 0.8170<jupyter_text>Explique aquí DropoutDropout significa "ignorar" algunas neuronas al azar (cantidad de acuerdo al dropout rate) para los pasos forward propagation o backward propagation. Esto ayuda a prevenir el overfitting, ya que la red se ve forzada a poder generalizar con un subconjunto de neuronas y a no desarrollar codependencias entre neuronas.<jupyter_code>loss, acc = model.evaluate(X_te_pad, y_test)
print('Accuracy en datos de test: {0}'.format(acc))<jupyter_output>3000/3000 [==============================] - 1s 460us/sample - loss: 0.4348 - accuracy: 0.8040
Accuracy en datos de test: 0.8040000200271606<jupyter_text>Red LSTM con Embedding<jupyter_code>document_input = Input(shape=(max_words, ), dtype='int32')
# Embedding Layer
embedding = Embedding(input_dim=num_words, output_dim=embedding_size, 
                      input_length=max_words, 
                      trainable=True)
embedding_output = embedding(document_input)

rnn = LSTM(10,dropout=0.4)(embedding_output)


output = Dense(num_classes, activation='softmax')(rnn)

# Construyendo el modelo
model = Model(inputs=[document_input], outputs=[output])
model.compile(loss='categorical_crossentropy', optimizer='adam',
             metrics=['accuracy'])

# Entrenando el modelo
history = model.fit(X_tr_pad, y_train,
                  batch_size=batch_size,
                  epochs=2,
                  verbose=1,
                  validation_split=0.2)
loss, acc = model.evaluate(X_te_pad, y_test)
print('Accuracy en datos de test: {0}'.format(acc))
%matplotlib inline
from matplotlib import pyplot as plt
fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(history.history['loss'])
ax.plot(history.history['val_loss'])
ax.set_xlabel('epoch')
ax.set_ylabel('loss')
fig.legend(['train', 'validation'], loc='upper left')<jupyter_output><empty_output><jupyter_text>Escriba aquí sus conclusionesEl error de validación se mantiene más o menos constante. Sin embargo, la accuracy (métrica que estamos optimizando) si aumenta.  Red LSTM con GLOVE embedding Descargue los embeddings pre entrenados desde https://nlp.stanford.edu/projects/glove/archivo glove.6B.zip y extraiga los datos para utilizarlos en esta sección<jupyter_code># Funciones para cargar los vectores GLOVE
def glorot_uniform_np(shape):
    fan_in, fan_out = shape[0],shape[1]
    s = np.sqrt(6. / (fan_in + fan_out))
    return np.random.uniform(-s, s, size=shape)

def load_word_vectors(embeddings_index, glove_file):
    print('Indexing word vectors.')
    for line in glove_file:
        values = line.split()
        word = values[0]
        coefs = np.asarray(values[1:], dtype='float32')
        embeddings_index[word] = coefs
    glove_file.close()
    print('Found %s word vectors.' % len(embeddings_index))    


# Cargando los vectores GLOVE
## LOAD PRETRAINED WORD VECTORS
GLOVE_FILE = 'glove.6B.100d.txt'
embeddings_index = {}
glove_file = open(GLOVE_FILE)
                  
load_word_vectors(embeddings_index, glove_file)   
embedding_matrix = np.zeros((len(word_index) + 1, 100))
for word, i in word_index.items():
    embedding_vector = embeddings_index.get(word)
    if embedding_vector is not None:
        embedding_matrix[i] = embedding_vector
embeddings_index = {}
from tensorflow.keras.initializers import Constant

document_input = Input(shape=(max_words, ), dtype='int32')

embedding = Embedding(input_dim=num_words, output_dim=100, 
                      input_length=max_words, 
                      trainable=True,embeddings_initializer=Constant(embedding_matrix))
embedding_output = embedding(document_input)

f = Lambda(lambda x: K.sum(x,axis=1))(embedding_output)
y = Dense(20, activation='relu')(f)
output = Dense(num_classes, activation='softmax')(y)


# Construyendo el modelo
model = Model(inputs=[document_input], outputs=[output])
model.compile(loss='categorical_crossentropy', optimizer='adam',
             metrics=['accuracy'])

# Entrenando el modelo
history = model.fit(X_tr_pad, y_train,
                  batch_size=batch_size,
                  epochs=2,
                  verbose=1,
                  validation_split=0.2)
loss, acc = model.evaluate(X_te_pad, y_test)
print('Accuracy en datos de test: {0}'.format(acc))
rnn = SimpleRNN(10,return_sequences=False)(embedding_output)


# Aplicando una capa densa luego de la RNN
output = Dense(num_classes, activation='softmax')(rnn)

# Construyendo el modelo
model = Model(inputs=[document_input], outputs=[output])
model.compile(loss='categorical_crossentropy', optimizer='adam',
             metrics=['accuracy'])

# Entrenando el modelo
history = model.fit(X_tr_pad, y_train,
                  batch_size=batch_size,
                  epochs=2,
                  verbose=1,
                  validation_split=0.2)
loss, acc = model.evaluate(X_te_pad, y_test)
print('Accuracy en datos de test: {0}'.format(acc))
rnn = SimpleRNN(10,return_sequences=False)(embedding_output)


# Aplicando una capa densa luego de la RNN
output = Dense(num_classes, activation='softmax')(rnn)

# Construyendo el modelo
model = Model(inputs=[document_input], outputs=[output])
model.compile(loss='categorical_crossentropy', optimizer='adam',
             metrics=['accuracy'])

# Entrenando el modelo
history = model.fit(X_tr_pad, y_train,
                  batch_size=batch_size,
                  epochs=2,
                  verbose=1,
                  validation_split=0.2)
loss, acc = model.evaluate(X_te_pad, y_test)
print('Accuracy en datos de test: {0}'.format(acc))
rnn = SimpleRNN(10,dropout=0.4,return_sequences=False)(embedding_output)


# Aplicando una capa densa luego de la RNN
output = Dense(num_classes, activation='softmax')(rnn)

# Construyendo el modelo
model = Model(inputs=[document_input], outputs=[output])
model.compile(loss='categorical_crossentropy', optimizer='adam',
             metrics=['accuracy'])

# Entrenando el modelo
history = model.fit(X_tr_pad, y_train,
                  batch_size=batch_size,
                  epochs=2,
                  verbose=1,
                  validation_split=0.2)
loss, acc = model.evaluate(X_te_pad, y_test)
print('Accuracy en datos de test: {0}'.format(acc))
rnn = LSTM(10,dropout=0.4)(embedding_output)
output = Dense(num_classes, activation='softmax')(rnn)

# Construyendo el modelo
model = Model(inputs=[document_input], outputs=[output])
model.compile(loss='categorical_crossentropy', optimizer='adam',
             metrics=['accuracy'])

# Entrenando el modelo
history = model.fit(X_tr_pad, y_train,
                  batch_size=batch_size,
                  epochs=3,
                  verbose=1,
                  validation_split=0.2)
loss, acc = model.evaluate(X_te_pad, y_test)
print('Accuracy en datos de test: {0}'.format(acc))<jupyter_output>3000/3000 [==============================] - 2s 517us/sample - loss: 0.4771 - accuracy: 0.8097
Accuracy en datos de test: 0.8096666932106018<jupyter_text>Visualizando Words Embedding con PCA (opcional, 15 ptos.)Este ejercicio es más difícil, pero resolverlo aclarará muchas dudas sobre el funcionamiento de Keras.Visualice los embeddings entrenados utilizando PCA.Para esto debe seguir los siguientes pasos:1. Obtenga los pesos de la capa de embedding, estos tendran tamaño (num_vocab, embedding_dim). Puede obtener la correspondencia entre vectores y palabras utilizando el word_index.2. Ajuste un modelo PCA (https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html) y quédese con los dos componentes principales.3. Visualice algunas palabras tomadas alreatoriamente en un gráfico de dos dimensiones y concluya.<jupyter_code>from sklearn.decomposition import PCA
import random
embedding_weights = embedding.get_weights()[0]
pca = PCA(n_components=2,svd_solver="full")
pca.fit(embedding_weights)
new_embeddings = pca.transform(embedding_weights)
words_embeddings = {w:new_embeddings[idx] for w, idx in word_index.items()}
fig = plt.figure(figsize=(15,15), facecolor='white')
ax = fig.add_subplot(111)
cont = 0
random.seed(0)
while(cont < 20):
    random_number = random.randint(0,len(word_index.items()))
    palabra_random = list(words_embeddings.keys())[random_number]
    ax.scatter(words_embeddings[palabra_random][0],words_embeddings[palabra_random][1], 
             marker='o',label=palabra_random)
    ax.annotate(palabra_random, (words_embeddings[palabra_random][0],words_embeddings[palabra_random][1]))
    cont += 1
ax.set_xlabel('component 1')
ax.set_ylabel('component 2')
ax.legend()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11731843575418995, 'fraction_numerical': 0.02669149596523898, 'mean_word_length': 4.486189935679152, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 41, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '172540', 'license': 'Apache-2.0', 'n_tokens_mistral': 5150, 'n_tokens_neox': 4777, 'n_words': 1055, 'path': 'Aprendizaje Supervisado/Otros/imdb_sentiment_deep_hw.ipynb', 'repo_name': 'franciscoandrades/Portafolio', 'size': '14511'}