<jupyter_start><jupyter_text>This tutorial guides you on how to train a word embedding from scratch and visualize the embeddings using the Embedding Projector.<jupyter_code>!pip install -q tf-nightly tfds-nightly
import tensorflow as tf
import tensorflow_datasets as tfds
import matplotlib.pyplot as plt
import io

tfds.disable_progress_bar()

print("Tensorflow Version: {}".format(tf.__version__))
print("GPU {} available.".format("is" if tf.config.experimental.list_physical_devices("GPU") else "not"))<jupyter_output>Tensorflow Version: 2.2.0-dev20200211
GPU is available.<jupyter_text>Representing Text as NumbersThe machine learning model takes numbers as inputs. When it comes to the text, the first thing is to convert the text into numbers or to vectorize the text.  One-hot Encodings![One-hot Encoding](https://github.com/tensorflow/docs/blob/master/site/en/tutorials/text/images/one-hot.png?raw=1)from tensorflow.org (2020)One-hot encoding is inefficient and sparse (most numbers in the vectors are zeros).  Encode Each Word with a Unique NumberThe example of the encoding using this approach is that `The cat is sitting on the car.` is converted to `[6, 2, 3, 5, 4, 6, 1]`.This approach is efficient and dense, however, it has several downsides.* The integer-encoding is arbitrary. (it can't show any relationships between words)* It is hard to interpret the model. **Because there is no relationship between the similarity of any two words and the similarity of their encodings, this feature-weight combination is not meaningful.**  Word EmbeddingsWord embeddings provide an efficient and dense representation way to show the relationships between words (similar words have similar vectors). More importantly, the word embedding layer is the byproduct of the training task to a specific purpose. When we train a text model on the final layer, we also train the embedding layer on the previous layers as well. One of the previous layers is the embedding layer. It is common to see word embeddings in 8 dimensions (for a small dataset), up to 1024 dimensions (for a large dataset). However, more dimensions require more datasets to train.The below is the example. Each word is represented as a 4-dimensional vector of floating values.![](https://github.com/tensorflow/docs/blob/master/site/en/tutorials/text/images/embedding2.png?raw=1)Tensorflow.org(2020)Another concept to think of a word embedding is as `lookup table`.  Using the Embedding Layer The embedding layer can be thought of as a lookup table that maps from integer indices (stand for a group of words) to dense vectors. The `input_dim` value is the size of the total words (more specific the word in an encoder). The `output_dim` value is the value that you can experiment with, like the value of `units` in the Dense layer.<jupyter_code># each word is represented as 5-dimensional vector of floating values
embedding_layer = tf.keras.layers.Embedding(input_dim=1000, output_dim=5)<jupyter_output><empty_output><jupyter_text>If you pass the integer to an embedding layer, that means lookup the table for the vectors representing the group of integers (or words). If an embedding layer was created, the weight values were also randomly generated. After training, the weight value would be updated that is like the operation of the other layers. The lookup index can't be assigned over 1000 (because the `input_dim` was set to 1000, representing 1000 words).<jupyter_code>result = embedding_layer(tf.constant([1, 2, 3, 999]))
result.numpy()<jupyter_output><empty_output><jupyter_text>For the text or sequence problems, the embedding layer takes 2D tensor of integer, whose shapes are `(sample, sequence_length)`. Each entry is a sequence of integers. The returned tensor has one more axis than the input tensor, the returned tensor's shape becomes `(sample, sequence_length, embedding_output_dim)`. Such output tensor's shape is variant if you try to convert these sequences to a fixed representation for further connections (like dense layers), there are several approaches. For example, you can use RNN, Attention, or a pooling layer (e.g. GlobalAveragePooling1D(), etc.) before passing it to a Dense layer.<jupyter_code>inputs = tf.constant([[1, 2, 3], [10, 20, 30]])
print(inputs.shape)

result = embedding_layer(inputs)
print(result.shape)<jupyter_output>(2, 3)
(2, 3, 5)<jupyter_text>Learning Embeddings from Scratch Here we are going to train a sentiment classifier on the IMDB dataset for training an embedding layer from scratch.<jupyter_code>(train_data, test_data), info = tfds.load(
  name="imdb_reviews/subwords8k",
  split=(tfds.Split.TRAIN, tfds.Split.TEST),
  with_info=True,
  as_supervised=True)<jupyter_output><empty_output><jupyter_text>Before we start training, let's get a decoder and have a look at the vocabulary that the decoder encodes.<jupyter_code>encoder = info.features['text'].encoder
encoder.subwords[:20]<jupyter_output><empty_output><jupyter_text>Movie reviews can be of different lengths. Here we use the `padded_batch` method to standardize the sequence length.<jupyter_code>train_batches = train_data.shuffle(1000).padded_batch(10)
test_batches = test_data.shuffle(1000).padded_batch(10)
train_batch, train_labels = next(iter(train_batches))
train_batch.numpy()<jupyter_output><empty_output><jupyter_text>Create a Simple ModelNext, we are going to build a model. In this case, it is a `Continuous Bags of Words` style of model. Here we use the `GlobalAveragePooling1D()` method to handle the length variable issue.<jupyter_code>embedding_dims = 16

def build_model():
  def _model(inputs):
    embed = tf.keras.layers.Embedding(
      input_dim=encoder.vocab_size, output_dim=embedding_dims)(inputs)
    ave = tf.keras.layers.GlobalAveragePooling1D()(embed)
    x = tf.keras.layers.Dense(units=16, activation='elu')(ave)
    classifier = tf.keras.layers.Dense(units=1, activation='sigmoid')(x)
    return classifier

  inputs = tf.keras.Input(shape=(None,))
  classifier = _model(inputs)
  return tf.keras.Model(inputs, classifier)

model = build_model()
model.summary()<jupyter_output>Model: "model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, None)]            0         
_________________________________________________________________
embedding_1 (Embedding)      (None, None, 16)          130960    
_________________________________________________________________
global_average_pooling1d (Gl (None, 16)                0         
_________________________________________________________________
dense (Dense)                (None, 16)                272       
_________________________________________________________________
dense_1 (Dense)              (None, 1)                 17        
=================================================================
Total params: 131,249
Trainable params: 131,249
Non-trainable params: 0
_______________________________________________________[...]<jupyter_text>Compile and Train the Model<jupyter_code>model.compile(loss=tf.keras.losses.BinaryCrossentropy(),
              optimizer=tf.keras.optimizers.Adam(),
              metrics=[tf.keras.metrics.BinaryAccuracy()])
history = model.fit(train_batches, 
                    epochs=10, 
                    validation_data=test_batches, 
                    validation_steps=20)
his = history.history
print(his.keys())
plt.figure(figsize=(14, 6))

plt.subplot(1, 2, 1)
plt.plot(his['loss'], 'bo', label='Training')
plt.plot(his['val_loss'], 'r', label='Validation')
plt.title("Loss")
plt.legend(loc='upper left')

plt.subplot(1, 2, 2)
plt.plot(his['binary_accuracy'], 'bo', label='Training')
plt.plot(his['val_binary_accuracy'], 'r', label="Validation")
plt.title('Accuracy')
plt.ylim([0.5, 1.])
plt.legend(loc='lower right')

plt.show()<jupyter_output><empty_output><jupyter_text>Let's look at the training history, the overfitting appears.  Retrieve the Learned Embeddings<jupyter_code>embed = model.get_layer('embedding_1')
embed_weights = embed.get_weights()[0]
embed_weights.shape<jupyter_output><empty_output><jupyter_text>You can also use the index to get the layer.<jupyter_code>embed_index = model.layers[1]
embed_index_weights = embed_index.get_weights()[0]
embed_index_weights.shape<jupyter_output><empty_output><jupyter_text>Next, write the weights to the disk.<jupyter_code>encoder = info.features['text'].encoder

out_v = io.open('vecs.tsv', 'w', encoding='utf-8')
out_m = io.open('meta.tsv', 'w', encoding='utf-8')

for num, word in enumerate(encoder.subwords):
  vec = embed_weights[num+1]  # skip 0, it is padding
  out_m.write(word + "\n")
  out_v.write('\t'.join([str(x) for x in vec]) + "\n")

out_v.close()
out_m.close()<jupyter_output><empty_output><jupyter_text>Download the weights into the local.<jupyter_code>try:
  from google.colab import files
except ImportError:
  pass
else:
  files.download("vecs.tsv")
  files.download("meta.tsv")<jupyter_output><empty_output><jupyter_text>Visualize the Embeddings<jupyter_code>from sklearn.manifold import TSNE
import numpy as np
tsne = TSNE(n_components=2, perplexity=30.0, init='pca', n_iter=5000)
plot_embedding = np.asfarray(embed_weights[:1000, :], dtype='float')
low_dim_embs = tsne.fit_transform(plot_embedding)
labels = encoder.subwords[:1000]
plt.figure(figsize=(20, 20))

for i, label in enumerate(labels):
  x, y = low_dim_embs[i, :]
  plt.scatter(x, y)
  plt.annotate(label, xy=(x, y), xytext=(5, 2), textcoords='offset points',
               ha='right', va='bottom')

plt.show()<jupyter_output>/usr/local/lib/python3.6/dist-packages/matplotlib/backends/backend_agg.py:211: RuntimeWarning: Glyph 150 missing from current font.
  font.set_text(s, 0.0, flags=flags)
/usr/local/lib/python3.6/dist-packages/matplotlib/backends/backend_agg.py:180: RuntimeWarning: Glyph 150 missing from current font.
  font.set_text(s, 0, flags=flags)<jupyter_text>You can also use the tool [Embedding Projector](http://projector.tensorflow.org/) to visualize the embedding layer.<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11130742049469965, 'fraction_numerical': 0.019532783667059284, 'mean_word_length': 5.332504661280298, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '453353', 'license': 'MIT', 'n_tokens_mistral': 3183, 'n_tokens_neox': 2947, 'n_words': 954, 'path': 'deep_learning/time_series_sequences/TF2Keras_WordEmbeddings.ipynb', 'repo_name': 'jiankaiwang/sophia.ml', 'size': '10188'}