<jupyter_start><jupyter_text>Structure of notebook- This notebook will serve as an introduction to the word embedding process in Keras.- Here we introduce:    1. Tokenization    2. Embedding- The following notebook will show you how to:    1. Load a trained vector model    2. Use embedding for building a simple model  One hot encoding and text tokenizationHere we are going to tokenize our text and one-hot encode the words using keras's in-built tokenizer<jupyter_code>from keras.preprocessing.text import Tokenizer
import pandas as pd
import numpy as np
samples = ['The quick brown fox.', 'Jumped over the lazy fox.']
# Creates a tokenizer, configured to only take into account the 1000 most common words
# Note that we only have 7
tokenizer = Tokenizer(num_words = 1000)
# Building the word index
tokenizer.fit_on_texts(samples)
# Turns strings into lists of integer indices
sequences = tokenizer.texts_to_sequences(samples)
sequences
# Turns string into binary vector of of dim 1000 (based on word limit above)
one_hot_results = tokenizer.texts_to_matrix(samples, mode='binary')
pd.DataFrame(one_hot_results)
# Dictionary mapping of words to one-hot-encoded index value
word_index = tokenizer.word_index
print('Found {} unique tokens'.format(len(word_index)))
print('The dictionary mapping of tokens is\n {}'.format(word_index))<jupyter_output>Found 7 unique tokens
The dictionary mapping of tokens is
 {'lazy': 7, 'jumped': 5, 'brown': 4, 'fox': 2, 'over': 6, 'the': 1, 'quick': 3}<jupyter_text>Hashing trick- For very large vacabularies one-hot-encoding will not work. - Rather we use one-hot hashing which uses a light-weight hashing function to hash words into vectors of fixed size (rather than maintaing an index).- Advantages:    - Do not need to maintain word index    - Saves memory    - Allows online encoding of data (can generate token vectors on the fly before all data has been seen)- Disadvantages:    - Hash-collisions (occurs when 2 words occupt the same hash)- Practicalities:    - If the dimensionality of the hash-space is large then hash-collisions are unlikely<jupyter_code># Example of one-hot hashing
samples = ['The quick brown fox.', 'Jumped over the lazy fox.']
dimensionality = 10
max_length = 10

# Pre-allocation
results = np.zeros((len(samples), max_length, dimensionality))

# Hashing function 
for i, sample in enumerate(samples):
    for j, word in list(enumerate(sample.split()))[:max_length]:
        # Hashes word into a random integer index between 0 and 1000
        index = abs(hash(word)) % dimensionality
        results[i, j, index] = 1
# 2 arrays with hashes
results<jupyter_output><empty_output><jupyter_text>Setting up an embedding layer.- The embedding layer takes in a 2d tensor of integers of the dimension `(samples, sequence_length)`     - It accept batches of size `samples` and     - The string will need to be either 0 padded or truncated to reach the `sequence_length`<jupyter_code>from keras.layers import Embedding
# Embedding takes two values
# Embedding:(n,d) = (max number of tokens, embedding dimension)
embedding_layer = Embedding(1000, 64)<jupyter_output><empty_output><jupyter_text>- Here we'll use the imdb dataset    - The `x` values are tokenized values of words    - The `y` values are the sentiment score- We'll restrict our voacbulary to the 10000 most popular words with and cut-off reviews after 20 words<jupyter_code># --- Loading librarires ---
from keras.datasets import imdb
from keras import preprocessing

# --- Setting up constants ---
# Number of words as features, we keep only the top most-common words
max_features = 1000
# Max number of words in a review (truncate the rest)
maxlen = 20
# --- Reading in in data ---
# Loads the data as lists of integers
(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words = max_features)
# What do our x_train, y_train look like?
print("The tokenized vector for the first review:")
print(pd.DataFrame(x_train[1]).head(10))
print("The sentiment for the first review" + str(pd.DataFrame(y_train[[1]])))

# --- Preprocessing data to pad/truncate sequences ---
# Turns the lists of integers into a 2d integer tensor of shape (samples, maxlen)
x_train = preprocessing.sequence.pad_sequences(x_train, maxlen = maxlen)
x_test = preprocessing.sequence.pad_sequences(x_test, maxlen = maxlen)
x_train.shape
x_train[1,:]<jupyter_output><empty_output><jupyter_text>Training a model and embedding layer- Let's now train the classifier and the weights from the embedding layer- Note:    - The Embedding layer weights, like all other weights in the network will be trained (e.g. with stochastic gradient descent)    - Word embeddings can be pretrained with w2v and use them as initial weights for the Embedding layer    - You can then make the weights static or trainable, depending on your preference- The model we will train will be a single dense layer on top for classification     - This is equivalent to a simple logisitic regression    - We do not consider inter-word relationships    - Recurrent nets take into account word relations<jupyter_code># Load libraries
from keras.models import Sequential
from keras.layers import Flatten, Dense

# Setting up keras model
model = Sequential()

# Create embedding layer as input
# 1000 - number of words
# 8 - embedding dimension
# input_length = length of phrase
model.add(Embedding(10000,8, input_length = maxlen))
model.add(Flatten())

# Add sigmoid layer
model.add(Dense(1, activation = 'sigmoid'))
model.compile(optimizer = 'rmsprop', loss='binary_crossentropy', metrics=['acc'])
model.summary()


model.fit(x_train, y_train,
                   epochs = 5,
                   batch_size = 32,
                   validation_split = 0.2)<jupyter_output>Train on 20000 samples, validate on 5000 samples
Epoch 1/5
20000/20000 [==============================] - 1s 49us/step - loss: 0.6772 - acc: 0.6006 - val_loss: 0.6448 - val_acc: 0.6704
Epoch 2/5
20000/20000 [==============================] - 1s 40us/step - loss: 0.5830 - acc: 0.7188 - val_loss: 0.5629 - val_acc: 0.7046
Epoch 3/5
20000/20000 [==============================] - 1s 39us/step - loss: 0.5152 - acc: 0.7464 - val_loss: 0.5362 - val_acc: 0.7208
Epoch 4/5
20000/20000 [==============================] - 1s 41us/step - loss: 0.4879 - acc: 0.7607 - val_loss: 0.5299 - val_acc: 0.7292
Epoch 5/5
20000/20000 [==============================] - 1s 38us/step - loss: 0.4731 - acc: 0.7694 - val_loss: 0.5290 - val_acc: 0.7334
{'fraction_non_alphanumeric': 0.1090458488228005, 'fraction_numerical': 0.04120198265179678, 'mean_word_length': 4.318780889621087, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '120080', 'license': 'MIT', 'n_tokens_mistral': 2129, 'n_tokens_neox': 1886, 'n_words': 764, 'path': 'Materials/L6 - Unsupervised Methods in DL/learning_word_embeddings_imdb_classification.ipynb', 'repo_name': 'WuZhuoran/ANLY-590-Neural-Nets-and-Deep-Learning', 'size': '6456'}