<jupyter_start><jupyter_text>WORD EMBEDDINGSone hot vectors are wasteful of space, and can not scale to very large differences in values. word embeddings are associating every word with a float vector. This allows us to represent a very large variety of words with much less space (quasi logarithmic)A good word embedding should show structure shown in the words and their relationships.Example: If (dog, wolf, book, notes) are in the dataset, their vectors in n-D space should be representing similarity.dog and wolf should have less euclidean distance than suppose say dog and book. similarly book and notes should have less distance. Initially, the Embeddings are assigned randomly, and with training process, we use backpropagation to get better embeddings. If we are working on a document task, and we can get a good word embeddings trained on millions of sentences, we should use that.<jupyter_code>from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Flatten, Dense, Embedding

model = Sequential()

model.add(Embedding(max_features, 8, input_length=max_length))
model.add(Flatten())
model.add(Dense(1, activation='sigmoid'))

model.compile(
    optimizer='rmsprop',
    loss='binary_crossentropy',
    metrics=['acc']
)
model.summary()
history = model.fit(X_train, Y_train, 
    epochs=10, 
    batch_size=32, 
    validation_split=0.2
)<jupyter_output>Epoch 1/10
625/625 [==============================] - 2s 3ms/step - loss: 0.6704 - acc: 0.6173 - val_loss: 0.6210 - val_acc: 0.6948
Epoch 2/10
625/625 [==============================] - 2s 3ms/step - loss: 0.5420 - acc: 0.7523 - val_loss: 0.5254 - val_acc: 0.7322
Epoch 3/10
625/625 [==============================] - 2s 3ms/step - loss: 0.4616 - acc: 0.7837 - val_loss: 0.5010 - val_acc: 0.7480
Epoch 4/10
625/625 [==============================] - 2s 2ms/step - loss: 0.4240 - acc: 0.8066 - val_loss: 0.4946 - val_acc: 0.7558
Epoch 5/10
625/625 [==============================] - 2s 3ms/step - loss: 0.3984 - acc: 0.8202 - val_loss: 0.4955 - val_acc: 0.7592
Epoch 6/10
625/625 [==============================] - 2s 3ms/step - loss: 0.3776 - acc: 0.8310 - val_loss: 0.4994 - val_acc: 0.7572
Epoch 7/10
625/625 [==============================] - 2s 2ms/step - loss: 0.3594 - acc: 0.8422 - val_loss: 0.5021 - val_acc: 0.7590
Epoch 8/10
625/625 [==============================] - 1s 2ms/step - loss: 0.[...]<jupyter_text>This can easily be increased upto around 90% accuracy by just increasing max_length = 200 words instead. But since this network has only one Dense neuron, it is essentially weighted sum of everything before. sum(x(i) * w(i)) does not take into account relationships between words<jupyter_code>from os.path import join as pjoin

raw = "aclImdb"
base_dir = pjoin(os.getcwd(), raw)
train_dir = pjoin(base_dir, 'train')
test_dir = pjoin(base_dir, 'test')

labels = []
texts = []

for label_type in ['neg', 'pos']:
    dirname = pjoin(train_dir, label_type)
    for filename in os.listdir(dirname):
        if filename[-4:] == '.txt':
            contents = open(pjoin(dirname, filename),encoding='utf-8').read()
            texts.append(contents)
            labels.append(1 if label_type == 'pos' else 0)
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
import numpy as np 

maxlen = 100
training_samples = 200
validation_samples = 10000
max_words = 10000


tokenizer = Tokenizer(num_words=max_words)
# assign number to each word
tokenizer.fit_on_texts(texts)
# convert every sentence into an array of numbers
sequences = tokenizer.texts_to_sequences(texts)
print(f"Found {len(tokenizer.word_index)} words")
data = pad_sequences(sequences, maxlen=maxlen)
labels = np.asarray(labels)
data.shape, labels.shape
# We will train on only 200 samples, and use a pretrained word embedding to get good performance
indices = np.arange(data.shape[0])
np.random.shuffle(indices)

data = data[indices]
labels = labels[indices]

X_train = data[:training_samples]
Y_train = labels[:training_samples]

X_val = data[training_samples: training_samples + validation_samples]
Y_val = labels[training_samples: training_samples + validation_samples]
# Download the pretrained word embedding -> https://nlp.stanford.edu/projects/glove/
glove_dir = pjoin(os.getcwd(), 'glove.6B')
embeddings_index = {}

with open(pjoin(glove_dir, "glove.6B.100d.txt"), encoding='utf-8') as reader:
    for line in reader:
        values = line.split()
        word = values[0]
        coefficients = values[1:]
        embeddings_index[word] = coefficients

print(f"Found {len(embeddings_index)} word vectors")

word_index = tokenizer.word_index
embedding_dim = 100

embedding_matrix = np.zeros((max_words, embedding_dim))
for word, i in word_index.items():
    if i < max_words:
        embedding_vector = embeddings_index.get(word)
        if embedding_vector is not None:
            embedding_matrix[i] = embedding_vector

model = Sequential()
model.add(Embedding(max_words, embedding_dim, input_length=maxlen))
model.add(Flatten())
model.add(Dense(32, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
model.summary()

model.layers[0].set_weights([embedding_matrix])
model.layers[0].trainable = False
model.compile(
    optimizer='rmsprop',
    loss='binary_crossentropy',
    metrics=['acc']
)
history = model.fit(
    X_train, Y_train, 
    epochs=10, 
    batch_size=32, 
    validation_data=(X_val, Y_val)
)<jupyter_output>Epoch 1/10
7/7 [==============================] - 1s 95ms/step - loss: 2.0398 - acc: 0.4800 - val_loss: 0.7291 - val_acc: 0.4967
Epoch 2/10
7/7 [==============================] - 1s 89ms/step - loss: 0.5055 - acc: 0.7950 - val_loss: 1.2128 - val_acc: 0.4995
Epoch 3/10
7/7 [==============================] - 1s 89ms/step - loss: 0.5465 - acc: 0.7150 - val_loss: 0.7197 - val_acc: 0.5558
Epoch 4/10
7/7 [==============================] - 1s 84ms/step - loss: 0.3306 - acc: 0.8650 - val_loss: 0.7516 - val_acc: 0.5566
Epoch 5/10
7/7 [==============================] - 1s 89ms/step - loss: 0.2502 - acc: 0.9150 - val_loss: 0.7906 - val_acc: 0.5509
Epoch 6/10
7/7 [==============================] - 1s 95ms/step - loss: 0.1225 - acc: 1.0000 - val_loss: 1.0304 - val_acc: 0.5213
Epoch 7/10
7/7 [==============================] - 1s 91ms/step - loss: 0.1437 - acc: 0.9750 - val_loss: 1.3921 - val_acc: 0.5008
Epoch 8/10
7/7 [==============================] - 1s 85ms/step - loss: 0.1404 - acc: 0.9550 - val[...]
{'fraction_non_alphanumeric': 0.1855161787365177, 'fraction_numerical': 0.07704160246533127, 'mean_word_length': 4.285830618892508, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '509901', 'license': 'MIT', 'n_tokens_mistral': 2523, 'n_tokens_neox': 2147, 'n_words': 617, 'path': 'books/DeepLearningWithKeras/RNN/WordEncoding.ipynb', 'repo_name': 'shibasis0801/MachineLearningPractice', 'size': '6490'}