<jupyter_start><jupyter_text>Embedding VisualizationIn Tensorflow, data is represented by __tensors__ in our graph. Tensors are representetives for high dimensional data. For example MNIST images have $28\times28=784$ dimensions, which are points in $\mathbb{R}^{784}$ space. As humans, we are able to process our world which is a $\mathbb{R}^{3}$ space ($x, y, z$). So, interpretation of the data is very hard in high dimensions.__Embeddin Visualization__ toolkit in Tensoarboard gives us the ability to reduce the dimension of our tensors using __Principle Component Analysis (PCA)__ or __t-Distributed Stochastic Neighbor Embedding (t-SNE)__ and visualize our data in lower dimension space.Let's build our Neural Net and try to visualize the tensors of activation of hidden layer.  Imports:We will start with importing the needed libraries for our code.<jupyter_code># imports
import os
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Input data:For this tutorial we use the MNIST dataset. MNIST is a dataset of handwritten digits. If you are into machine learning, you might have heard of this dataset by now. MNIST is kind of benchmark of datasets for deep learning. One other reason that we use the MNIST is that it is easily accesible through Tensorflow. If you want to know more about the MNIST dataset you can check Yann Lecun's website.We can easily import the dataset and see the size of training, test and validation set:<jupyter_code># Import MNIST data
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)

print("Size of:")
print("- Training-set:\t\t{}".format(len(mnist.train.labels)))
print("- Test-set:\t\t{}".format(len(mnist.test.labels)))
print("- Validation-set:\t{}".format(len(mnist.validation.labels)))<jupyter_output>Extracting MNIST_data/train-images-idx3-ubyte.gz
Extracting MNIST_data/train-labels-idx1-ubyte.gz
Extracting MNIST_data/t10k-images-idx3-ubyte.gz
Extracting MNIST_data/t10k-labels-idx1-ubyte.gz
Size of:
- Training-set:		55000
- Test-set:		10000
- Validation-set:	5000<jupyter_text>Hyper-parameters:Hyper-parameters are important parameters which are not learned by the network. So, we have to specify them externally. These parameters are constant and they are not learnable.<jupyter_code># hyper-parameters
logs_path = "./logs/embedding/"  # path to the folder that we want to save the logs for Tensorboard
learning_rate = 0.001  # The optimization learning rate
epochs = 10  # Total number of training epochs
batch_size = 100  # Training batch size
display_freq = 100  # Frequency of displaying the training results

# Network Parameters
# We know that MNIST images are 28 pixels in each dimension.
img_h = img_w = 28

# Images are stored in one-dimensional arrays of this length.
img_size_flat = img_h * img_w

# Number of classes, one class for each of 10 digits.
n_classes = 10

# number of units in the first hidden layer
h1 = 200<jupyter_output><empty_output><jupyter_text>Graph:Like before, we start by constructing the graph. But, we need to define some functions that we need rapidly in our code.<jupyter_code># weight and bais wrappers
def weight_variable(name, shape):
    """
    Create a weight variable with appropriate initialization
    :param name: weight name
    :param shape: weight shape
    :return: initialized weight variable
    """
    initer = tf.truncated_normal_initializer(stddev=0.01)
    return tf.get_variable('W_' + name,
                           dtype=tf.float32,
                           shape=shape,
                           initializer=initer)


def bias_variable(name, shape):
    """
    Create a bias variable with appropriate initialization
    :param name: bias variable name
    :param shape: bias variable shape
    :return: initialized bias variable
    """
    initial = tf.constant(0., shape=shape, dtype=tf.float32)
    return tf.get_variable('b_' + name,
                           dtype=tf.float32,
                           initializer=initial)


def fc_layer(x, num_units, name, use_relu=True):
    """
    Create a fully-connected layer
    :param x: input from previous layer
    :param num_units: number of hidden units in the fully-connected layer
    :param name: layer name
    :param use_relu: boolean to add ReLU non-linearity (or not)
    :return: The output array
    """
    with tf.variable_scope(name):
        in_dim = x.get_shape()[1]
        W = weight_variable(name, shape=[in_dim, num_units])
        tf.summary.histogram('W', W)
        b = bias_variable(name, [num_units])
        tf.summary.histogram('b', b)
        layer = tf.matmul(x, W)
        layer += b
        if use_relu:
            layer = tf.nn.relu(layer)
        return layer<jupyter_output><empty_output><jupyter_text>Now that we have our helper functions we can create our graph.<jupyter_code># Create graph
# Placeholders for inputs (x), outputs(y)
with tf.variable_scope('Input'):
    x = tf.placeholder(tf.float32, shape=[None, img_size_flat], name='X')
    tf.summary.image('input_image', tf.reshape(x, (-1, img_w, img_h, 1)), max_outputs=5)
    y = tf.placeholder(tf.float32, shape=[None, n_classes], name='Y')
fc1 = fc_layer(x, h1, 'Hidden_layer', use_relu=True)
output_logits = fc_layer(fc1, n_classes, 'Output_layer', use_relu=False)

# Define the loss function, optimizer, and accuracy
with tf.variable_scope('Train'):
    with tf.variable_scope('Loss'):
        loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y, logits=output_logits), name='loss')
        tf.summary.scalar('loss', loss)
    with tf.variable_scope('Optimizer'):
        optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate, name='Adam-op').minimize(loss)
    with tf.variable_scope('Accuracy'):
        correct_prediction = tf.equal(tf.argmax(output_logits, 1), tf.argmax(y, 1), name='correct_pred')
        accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32), name='accuracy')
        tf.summary.scalar('accuracy', accuracy)
        # Network predictions
        cls_prediction = tf.argmax(output_logits, axis=1, name='predictions')

# Initializing the variables
init = tf.global_variables_initializer()
merged = tf.summary.merge_all()<jupyter_output><empty_output><jupyter_text>Session:<jupyter_code># Launch the graph (session)
sess = tf.InteractiveSession() # using InteractiveSession instead of Session to test network in separate cell
sess.run(init)
train_writer = tf.summary.FileWriter(logs_path, sess.graph)
num_tr_iter = int(mnist.train.num_examples / batch_size)
global_step = 0
for epoch in range(epochs):
    print('Training epoch: {}'.format(epoch + 1))
    for iteration in range(num_tr_iter):
        batch_x, batch_y = mnist.train.next_batch(batch_size)
        global_step += 1
        # Run optimization op (backprop)
        feed_dict_batch = {x: batch_x, y: batch_y}
        _, summary_tr = sess.run([optimizer, merged], feed_dict=feed_dict_batch)
        train_writer.add_summary(summary_tr, global_step)

        if iteration % display_freq == 0:
            # Calculate and display the batch loss and accuracy
            loss_batch, acc_batch = sess.run([loss, accuracy],
                                             feed_dict=feed_dict_batch)
            print("iter {0:3d}:\t Loss={1:.2f},\tTraining Accuracy={2:.01%}".
                  format(iteration, loss_batch, acc_batch))

    # Run validation after every epoch
    feed_dict_valid = {x: mnist.validation.images, y: mnist.validation.labels}
    loss_valid, acc_valid = sess.run([loss, accuracy], feed_dict=feed_dict_valid)
    print('---------------------------------------------------------')
    print("Epoch: {0}, validation loss: {1:.2f}, validation accuracy: {2:.01%}".
          format(epoch + 1, loss_valid, acc_valid))
    print('---------------------------------------------------------')<jupyter_output>Training epoch: 1
iter   0:	 Loss=2.28,	Training Accuracy=37.0%
iter 100:	 Loss=0.45,	Training Accuracy=85.0%
iter 200:	 Loss=0.21,	Training Accuracy=95.0%
iter 300:	 Loss=0.34,	Training Accuracy=89.0%
iter 400:	 Loss=0.29,	Training Accuracy=90.0%
iter 500:	 Loss=0.18,	Training Accuracy=97.0%
---------------------------------------------------------
Epoch: 1, validation loss: 0.21, validation accuracy: 94.1%
---------------------------------------------------------
Training epoch: 2
iter   0:	 Loss=0.14,	Training Accuracy=97.0%
iter 100:	 Loss=0.19,	Training Accuracy=94.0%
iter 200:	 Loss=0.11,	Training Accuracy=95.0%
iter 300:	 Loss=0.06,	Training Accuracy=99.0%
iter 400:	 Loss=0.08,	Training Accuracy=96.0%
iter 500:	 Loss=0.12,	Training Accuracy=97.0%
---------------------------------------------------------
Epoch: 2, validation loss: 0.14, validation accuracy: 96.0%
---------------------------------------------------------
Training epoch: 3
iter   0:	 Loss=0.16,	Training Accuracy=96[...]<jupyter_text>Create embeddingsLet's say we are interested in visualizing the tensor of activation of the hidden layer nodes in the test test.In this example our test set has 10000 samples. Our hidden layer has $200$ nodes. So, the output tensor of hidden layer has a shape of 10000$\times$200.We will create an embedding variable with the shape (10000 , 200) and assing the of activation of the hidden layer ```(fc1)``` to the variable.<jupyter_code># Load the test set
x_test = mnist.test.images
y_test = mnist.test.labels

# Initialize the embedding variable with the shape of our desired tensor
tensor_shape = (x_test.shape[0] , fc1.get_shape()[1].value) # [test_set , h1] = [10000 , 200]
embedding_var = tf.Variable(tf.zeros(tensor_shape), 
                            name='fc1_embedding')
# assign the tensor that we want to visualize to the embedding variable
embedding_assign = embedding_var.assign(fc1)<jupyter_output><empty_output><jupyter_text>Now it is time to call the embedding functions to write the information in the memory.Remember, our goal is to visualize a tensor of high dimensional data, in a low dimension to be understandable for human. In order to understand the corresponding image and label ofeach point in the low dimension, we have two files:1. __metadata.tsv__: to store index and label of each sample2. __sprite_image.png__: to store all the sample images in a very large image The tensor that we are trying to visualize is a 10000$\times$200. We have 10000 samples. So, we need a __metadata.tsv__ file that stores 10000 indices of the images and their corresponding labels. We will also need to create a large image with containing 10000 small MNIST images (with size 28$\times$28).Without introducing the __metadata__ and __sprite image__ our visualization is nothing but some points with indices:So we will add the __metadata__ and __sprite image__ information to our config file.We will create this files later.  But in the embedding config file, we have to specify the path to these files.<jupyter_code>from tensorflow.contrib.tensorboard.plugins import projector

# Create a config object to write the configuration parameters
config = projector.ProjectorConfig()

# Add embedding variable
embedding = config.embeddings.add()
embedding.tensor_name = embedding_var.name

# Link this tensor to its metadata file (e.g. labels) -> we will create this file later
embedding.metadata_path = 'metadata.tsv'

# Specify where you find the sprite. -> we will create this image later
embedding.sprite.image_path = 'sprite_images.png'
embedding.sprite.single_image_dim.extend([img_w, img_h])

# Write a projector_config.pbtxt in the logs_path.
# TensorBoard will read this file during startup.
projector.visualize_embeddings(train_writer, config)<jupyter_output><empty_output><jupyter_text>Let's run our session to evaluate the tensor for embedding and save the data.<jupyter_code># Run session to evaluate the tensor
x_test_fc1 = sess.run(embedding_assign, feed_dict={x: x_test})

# Save the tensor in model.ckpt file
saver = tf.train.Saver()
saver.save(sess, os.path.join(logs_path, "model.ckpt"), global_step)<jupyter_output><empty_output><jupyter_text>create metadata and sprite images<jupyter_code>def write_sprite_image(filename, images):
    """
        Create a sprite image consisting of sample images
        :param filename: name of the file to save on disk
        :param shape: tensor of flattened images
    """

    # Invert grayscale image
    images = 1 - images

    # Calculate number of plot
    n_plots = int(np.ceil(np.sqrt(images.shape[0])))

    # Make the background of sprite image
    sprite_image = np.ones((img_h * n_plots, img_w * n_plots))

    for i in range(n_plots):
        for j in range(n_plots):
            img_idx = i * n_plots + j
            if img_idx < images.shape[0]:
                img = images[img_idx]
                sprite_image[i * img_h:(i + 1) * img_h,
                j * img_w:(j + 1) * img_w] = img

    plt.imsave(filename, sprite_image, cmap='gray')
    print('Sprite image saved in {}'.format(filename))


def write_metadata(filename, labels):
    """
            Create a metadata file image consisting of sample indices and labels
            :param filename: name of the file to save on disk
            :param shape: tensor of labels
    """
    with open(filename, 'w') as f:
        f.write("Index\tLabel\n")
        for index, label in enumerate(labels):
            f.write("{}\t{}\n".format(index, label))

    print('Metadata file saved in {}'.format(filename))


# Reshape images from vector to matrix
x_test_images = np.reshape(np.array(x_test), (-1, img_w, img_h))
# Reshape labels from one-hot-encode to index
x_test_labels = np.argmax(y_test, axis=1)

write_sprite_image(os.path.join(logs_path, 'sprite_images.png'), x_test_images)
write_metadata(os.path.join(logs_path, 'metadata.tsv'), x_test_labels)<jupyter_output>Sprite image saved in ./logs/embedding/sprite_images.png
Metadata file saved in ./logs/embedding/metadata.tsv<jupyter_text>After we are finished the testing, we will close the session to free the memory.<jupyter_code># close the session after you are done with testing
sess.close()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11715776117298746, 'fraction_numerical': 0.021429578457634286, 'mean_word_length': 3.5529525032092426, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '403185', 'license': 'MIT', 'n_tokens_mistral': 4517, 'n_tokens_neox': 4199, 'n_words': 1406, 'path': '4_Tensorboard/Tutorials/3_embedding_visualize.ipynb', 'repo_name': 'tamzi/easy-tensorflow', 'size': '14186'}