<jupyter_start><jupyter_text>Import TensorFlow 2.x.<jupyter_code>try:
  %tensorflow_version 2.x
except:
  pass

import tensorflow as tf
import tensorflow.keras.layers as layers
import tensorflow.keras.models as models

import numpy as np
np.random.seed(7)

print(tf.__version__)<jupyter_output>2.2.0<jupyter_text>Import TensorFlow datasets.*   MNIST dataset<jupyter_code>import tensorflow_datasets as tfds<jupyter_output><empty_output><jupyter_text>Import TensorFlow addons.* Triplet semi-hard loss<jupyter_code>import tensorflow_addons as tfa<jupyter_output><empty_output><jupyter_text>Load MNIST dataset.  Load MNIST dataset.* train split* test split<jupyter_code>train_dataset, test_dataset = tfds.load(name="mnist", split=['train', 'test'], as_supervised=True)<jupyter_output>WARNING:absl:Dataset mnist is hosted on GCS. It will automatically be downloaded to your
local data directory. If you'd instead prefer to read directly from our public
GCS bucket (recommended if you're running on GCP), you can instead set
data_dir=gs://tfds-data/datasets.<jupyter_text>Normalize dataset images.<jupyter_code>def _normalize_image(image, label):
    image = tf.cast(image, tf.float32) / 255.
    return (image, label)<jupyter_output><empty_output><jupyter_text>Create dataset batches.<jupyter_code>buffer_size = 1024
batch_size = 32
train_dataset = train_dataset.shuffle(buffer_size).batch(batch_size)
train_dataset = train_dataset.map(_normalize_image)

test_dataset = test_dataset.batch(batch_size)
test_dataset = test_dataset.map(_normalize_image)<jupyter_output><empty_output><jupyter_text>Create the model.* No activation (or default linear activation) on last layer* L2 normalized embeddings.<jupyter_code>model = models.Sequential([
    layers.Conv2D(filters=128, kernel_size=2, padding='same', activation='relu', input_shape=(28,28,1)),
    layers.Conv2D(filters=128, kernel_size=2, padding='same', activation='relu'),
    layers.MaxPooling2D(pool_size=2),

    layers.Dropout(0.3),

    layers.Conv2D(filters=64, kernel_size=2, padding='same', activation='relu'),
    layers.Conv2D(filters=64, kernel_size=2, padding='same', activation='relu'),
    layers.MaxPooling2D(pool_size=2),

    layers.Dropout(0.3),

    layers.Conv2D(filters=32, kernel_size=2, padding='same', activation='relu'),
    layers.Conv2D(filters=32, kernel_size=2, padding='same', activation='relu'),
    layers.MaxPooling2D(pool_size=2),

    layers.Dropout(0.3),
    
    layers.Flatten(),
    layers.Dense(256, activation=None),                      # No activation on final dense layer
    layers.Lambda(lambda x: tf.math.l2_normalize(x, axis=1)) # L2 normalized embeddings
])<jupyter_output><empty_output><jupyter_text>Show model summary.<jupyter_code>model.summary()<jupyter_output>Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d (Conv2D)              (None, 28, 28, 128)       640       
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 28, 28, 128)       65664     
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 14, 14, 128)       0         
_________________________________________________________________
dropout (Dropout)            (None, 14, 14, 128)       0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 14, 14, 64)        32832     
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 14, 14, 64)        16448     
________________________________________________________[...]<jupyter_text>Train the model.  Compile the model.<jupyter_code>model.compile(optimizer=tf.keras.optimizers.Adam(0.001), loss=tfa.losses.TripletSemiHardLoss())<jupyter_output><empty_output><jupyter_text>Train the model.<jupyter_code>history = model.fit(train_dataset, epochs=10)<jupyter_output>Epoch 1/10
1875/1875 [==============================] - 41s 22ms/step - loss: 0.4033
Epoch 2/10
1875/1875 [==============================] - 41s 22ms/step - loss: 0.1773
Epoch 3/10
1875/1875 [==============================] - 41s 22ms/step - loss: 0.1323
Epoch 4/10
1875/1875 [==============================] - 41s 22ms/step - loss: 0.1134
Epoch 5/10
1875/1875 [==============================] - 41s 22ms/step - loss: 0.0995
Epoch 6/10
1875/1875 [==============================] - 41s 22ms/step - loss: 0.0942
Epoch 7/10
1875/1875 [==============================] - 41s 22ms/step - loss: 0.0850
Epoch 8/10
1875/1875 [==============================] - 41s 22ms/step - loss: 0.0806
Epoch 9/10
1875/1875 [==============================] - 41s 22ms/step - loss: 0.0731
Epoch 10/10
1875/1875 [==============================] - 41s 22ms/step - loss: 0.0729<jupyter_text>Evaluate the model.* Create the embeddings for the test dataset.<jupyter_code>embeddings = model.predict(test_dataset)<jupyter_output><empty_output><jupyter_text>Save the embeddings for visualization in the embedding projector.<jupyter_code>import io

np.savetxt("embeddings-vecs.tsv", embeddings, delimiter='\t')

meta_file = io.open('embeddings-meta.tsv', 'w', encoding='utf-8')
for image, labels in tfds.as_numpy(test_dataset):
    [meta_file.write(str(label) + "\n") for label in labels]
meta_file.close()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17273559011893871, 'fraction_numerical': 0.0636779505946935, 'mean_word_length': 5.538277511961723, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '384348', 'license': 'MIT', 'n_tokens_mistral': 1958, 'n_tokens_neox': 1690, 'n_words': 347, 'path': 'TripletLoss/MNIST/Semi-hard.ipynb', 'repo_name': 'ashishpatel26/Tensorflow2-Examples', 'size': '5465'}