<jupyter_start><jupyter_text>Variational Auto  Encoders- Reference: Adapted from the Keras example- Auto-Encoding Variational Bayes   https://arxiv.org/abs/1312.6114<jupyter_code>import tensorflow as tf
from keras.backend.tensorflow_backend import set_session
config = tf.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.7
set_session(tf.Session(config=config))
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm

from keras.layers import Input, Dense, Lambda, Flatten, Reshape, Conv2D, Conv2DTranspose
from keras.models import Model
from keras import backend as K
from keras import metrics
from keras.datasets import fashion_mnist<jupyter_output><empty_output><jupyter_text>Fashion MNIST<jupyter_code>(x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()
plt.figure(figsize=(16, 8))
for i in range(0, 18):
    plt.subplot(3, 6, i + 1)
    plt.imshow(x_train[i], cmap="gray")
    plt.axis("off")
plt.show()
x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.<jupyter_output><empty_output><jupyter_text>Standard full-connected VAE modelLet's define a VAE model with fully connected MLPs for the encoder and decoder networks.<jupyter_code>x_train_standard = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))
x_test_standard = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))
x_train_standard.shape, x_test_standard.shape<jupyter_output><empty_output><jupyter_text>Encoder<jupyter_code>original_dim = 784
latent_dim = 2
intermediate_dim = 256


def make_encoder(original_dim, intermediate_dim, latent_dim):
    x = Input(shape=(original_dim,))
    hidden = Dense(intermediate_dim, activation='relu')(x)
    z_mean = Dense(latent_dim)(hidden)
    z_log_var = Dense(latent_dim)(hidden)
    return Model(inputs=x, outputs=[z_mean, z_log_var],
                name="mlp_encoder")

    
encoder = make_encoder(original_dim, intermediate_dim, latent_dim)<jupyter_output><empty_output><jupyter_text>The VAE stochastic latent variableWe use the reparametrization trick to define a random variable z that is conditioned on the input image x as follows:$$ z \sim \mathcal{N}(\mu_z(x), \sigma_z(x)) $$The reparametrization tricks defines $z$ has follows:$$ z = \mu_z(x) + \sigma_z(x) \cdot \epsilon$$with:$$ \epsilon \sim \mathcal{N}(0, 1) $$This way the dependency to between $z$ and $x$ is deterministic and differentiable. The randomness of $z$ only stems from $\epsilon$ only for a given $x$.Note that in practice the output of the encoder network parameterizes $log(\sigma^2_z(x)$ instead of $\sigma_z(x)$. Taking the exponential of $log(\sigma^2_z(x)$ ensures the positivity of the standard deviation from the raw output of the network:<jupyter_code>def sampling_func(inputs):
    z_mean, z_log_var = inputs
    batch_size = K.shape(z_mean)[0]
    epsilon = K.random_normal(shape=(batch_size, latent_dim),
                              mean=0., stddev=1.)
    return z_mean + K.exp(z_log_var / 2) * epsilon


sampling_layer = Lambda(sampling_func, output_shape=(latent_dim,),
                        name="latent_sampler")<jupyter_output><empty_output><jupyter_text>Decoder<jupyter_code>def make_decoder(latent_dim, intermediate_dim, original_dim):
    decoder_input = Input(shape=(latent_dim,))
    x = Dense(intermediate_dim, activation='relu')(decoder_input)
    x = Dense(original_dim, activation='sigmoid')(x)
    return Model(decoder_input, x, name="mlp_decoder")


decoder = make_decoder(latent_dim, intermediate_dim, original_dim)<jupyter_output><empty_output><jupyter_text>By default the decoder outputs has random weights and output noise:<jupyter_code>random_z_from_prior = np.random.normal(loc=0, scale=1, size=(1, latent_dim))
generated = decoder.predict(random_z_from_prior)
plt.imshow(generated.reshape(28, 28), cmap=plt.cm.gray)
plt.axis('off');<jupyter_output><empty_output><jupyter_text>The generated image is completely univariate noise: there is no apparent spatial depenedencies between the pixel values. This reflects the lack of prior structure in the randomly initialized fully-connected decoder network. Let's now the plug the encoder and decoder via the stochastic latent variable $z$ to get the full VAE architecture. The loss function is the negative ELBO of the variational inference problem:<jupyter_code>def make_vae(input_shape, encoder, decoder, sampling_layer):
    # Build de model architecture by assembling the encoder,
    # stochastic latent variable and decoder:
    x = Input(shape=input_shape, name="input")
    z_mean, z_log_var = encoder(x)
    z = sampling_layer([z_mean, z_log_var])
    x_decoded_mean = decoder(z)
    vae = Model(x, x_decoded_mean)

    # Define the VAE loss
    xent_loss = original_dim * metrics.binary_crossentropy(
        K.flatten(x), K.flatten(x_decoded_mean))
    kl_loss = - 0.5 * K.sum(1 + z_log_var - K.square(z_mean) - K.exp(z_log_var), axis=-1)
    vae_loss = K.mean(xent_loss + kl_loss)

    vae.add_loss(vae_loss)
    vae.compile(optimizer='adam')
    return vae

vae = make_vae((original_dim,), encoder, decoder,
               sampling_layer=sampling_layer)
vae.summary()
vae.fit(x_train_standard, epochs=50, batch_size=100,
        validation_data=(x_test_standard, None))
# vae.save_weights("standard_weights.h5")
vae.load_weights("standard_weights.h5")<jupyter_output><empty_output><jupyter_text>Note that the model has not yet converged even after 50 epochs. Furthermore it's is not overfitting significantly either. We chose a very low value for the latent dimension. It is likely that using the higher dimensional space could lead to a model either to optimize that would better fit the training set.By sampling a random latent vector from the prior distribution and feeding it to the decoder we can effectively sample from the image model trained by the VAE:<jupyter_code>random_z_from_prior = np.random.normal(size=(1, latent_dim))
generated = decoder.predict(random_z_from_prior)
plt.imshow(generated.reshape(28, 28), cmap=plt.cm.gray)
plt.axis('off');<jupyter_output><empty_output><jupyter_text>Use `Ctrl-Enter` several times to sample from various random locations in the 2D latent space.The generated pictures are blurry but capture of the global organization of pixels required to represent samples from the 10 fashion item categories. The spatial structure has been learned and is only present in the decoder weights.  2D plot of the image classes in the latent spaceWe can also use the encoder to set the visualize the distribution of the test set in the 2D latent space of the VAE model. In the following the colors show the true class labels from the test samples.Note that the VAE is an unsupervised model: it did not use any label information during training. However we can observe that the 2D latent space is largely structured around the categories of images used in the training set.<jupyter_code>id_to_labels = {0: "T-shirt/top", 1: "Trouser", 2: "Pullover", 3: "Dress", 4: "Coat", 
                5: "Sandal", 6: "Shirt", 7: "Sneaker", 8: "Bag", 9: "Ankle boot"}
x_test_encoded, x_test_encoded_log_var = encoder.predict(x_test_standard, batch_size=100)
plt.figure(figsize=(7, 6))
plt.scatter(x_test_encoded[:, 0], x_test_encoded[:, 1], c=y_test,
            cmap=plt.cm.tab10)
cb = plt.colorbar()
cb.set_ticks(list(id_to_labels.keys()))
cb.set_ticklabels(list(id_to_labels.values()))
cb.update_ticks()
plt.show()<jupyter_output><empty_output><jupyter_text>**Exercises**- One can see that the class labels 5, 7 and 9 are grouped in a cluster of the latent space. Use matplotlib to display some samples from each of those 3 classes and discover why they have been grouped together by the VAE model.- Similarly: can you qualitatively explain with matplotlib why class 0, 4 and 6 seem to be hard to disentangle in this 2D latent space discovered by the VAE model?- One can observe that the global 2D shape of the encoded dataset is approximately spherical with values with a maximum radius of size 3. Where can you explain where the shape of this marginal latent distribution come from?<jupyter_code># %load solutions/class_5_7_9.py
# %load solutions/class_0_4_6.py
# %load solutions/shape_marginal_latent_distribution.py<jupyter_output><empty_output><jupyter_text>2D panel view of samples from the VAE manifoldThe following linearly spaced coordinates on the unit square were transformed through the inverse CDF (ppf) of the Gaussian to produce values of the latent variables z. This makes it possible to use a square arangement of panels that spans the gaussian prior of the latent space.<jupyter_code>n = 15  # figure with 15x15 panels
digit_size = 28
figure = np.zeros((digit_size * n, digit_size * n))
grid_x = norm.ppf(np.linspace(0.05, 0.95, n))
grid_y = norm.ppf(np.linspace(0.05, 0.95, n))

for i, yi in enumerate(grid_x):
    for j, xi in enumerate(grid_y):
        z_sample = np.array([[xi, yi]])
        x_decoded = decoder.predict(z_sample)
        digit = x_decoded[0].reshape(digit_size, digit_size)
        figure[i * digit_size: (i + 1) * digit_size,
               j * digit_size: (j + 1) * digit_size] = digit

plt.figure(figsize=(10, 10))
plt.imshow(figure, cmap='Greys_r')
plt.show()<jupyter_output><empty_output><jupyter_text>Convolutional Variational Auto Encoder<jupyter_code>x_train_conv = np.expand_dims(x_train, -1)
x_test_conv = np.expand_dims(x_test, -1)
x_train_conv.shape, x_test_conv.shape<jupyter_output><empty_output><jupyter_text>**Exercise**: write an encoder that uses a series of convolutional layers, with maxpooling or strided convolutions and Batch norm to encode the 2D, gray-level images into 2D latent vectors:<jupyter_code>from keras.layers import BatchNormalization

img_rows, img_cols, img_chns = 28, 28, 1
filters = 32
kernel_size = 3
intermediate_dim = 128
latent_dim = 2


def make_conv_encoder(img_rows, img_cols, img_chns,
                      latent_dim, intermediate_dim):
    inp = x = Input(shape=(img_rows, img_cols, img_chns))
    # TODO: write me!
    return Model(inputs=inp, outputs=[z_mean, z_log_var],
                 name='convolutional_encoder')


conv_encoder = make_conv_encoder(img_rows, img_cols, img_chns,
                                 latent_dim, intermediate_dim)
print(conv_encoder.summary())
# %load solutions/conv_encoder.py<jupyter_output><empty_output><jupyter_text>The stochastic latent variable is the same as for the fully-connected model.<jupyter_code>sampling_layer = Lambda(sampling_func, output_shape=(latent_dim,),
                        name="latent_sampler")<jupyter_output><empty_output><jupyter_text>DecoderThe decoder is also convolutional but instead of downsampling the spatial dimensions from (28, 28) to 2 latent dimensions, it starts from the latent space to upsample a (28, 28) dimensions using strided `Conv2DTranspose` layers.Here again BatchNormalization layers are inserted after the convolution to make optimization converge faster.<jupyter_code>def make_conv_decoder(latent_dim, intermediate_dim, original_dim,
                      spatial_size=7, filters=16):
    decoder_input = Input(shape=(latent_dim,))
    x = Dense(intermediate_dim, activation='relu')(decoder_input)
    x = Dense(filters * spatial_size * spatial_size, activation='relu')(x)
    x = Reshape((spatial_size, spatial_size, filters))(x)
    # First up-sampling:
    x = Conv2DTranspose(filters,
                        kernel_size=3,
                        padding='same',
                        strides=(2, 2),
                        activation='relu')(x)
    x = BatchNormalization()(x)
    x = Conv2DTranspose(filters,
                        kernel_size=3,
                        padding='same',
                        strides=1,
                        activation='relu')(x)
    x = BatchNormalization()(x)
    
    # Second up-sampling:
    x = Conv2DTranspose(filters,
                        kernel_size=3,
                        strides=(2, 2),
                        padding='valid',
                        activation='relu')(x)
    x = BatchNormalization()(x)
    # Ouput 1 channel of gray pixels values between 0 and 1:
    x = Conv2D(1, kernel_size=2, padding='valid',
               activation='sigmoid')(x)
    return Model(decoder_input, x, name='convolutional_decoder')


conv_decoder = make_conv_decoder(latent_dim, intermediate_dim, original_dim,
                                 spatial_size=7, filters=filters)
print(conv_decoder.summary())
generated = conv_decoder.predict(np.random.normal(size=(1, latent_dim)))
plt.imshow(generated.reshape(28, 28), cmap=plt.cm.gray)
plt.axis('off');<jupyter_output><empty_output><jupyter_text>This new decoder encodes some a priori knowledge on the local dependencies between pixel values in the "deconv" architectures. Depending on the randomly initialized weights, the generated images can show some local spatial structure.Try to re-execute the above two cells several times to try to see the kind of local structure that stem from the "deconv" architecture it-self for different random initializations of the weights.Again, let's now plug everything to together to get convolutional version of a full VAE model:<jupyter_code>input_shape = (img_rows, img_cols, img_chns)
vae = make_vae(input_shape, conv_encoder, conv_decoder,
               sampling_layer)
vae.summary()
vae.fit(x_train_conv, epochs=15, batch_size=100,
        validation_data=(x_test_conv, None))
# vae.save_weights("convolutional_weights.h5")
vae.load_weights("convolutional_weights.h5")
generated = conv_decoder.predict(np.random.normal(size=(1, latent_dim)))
plt.imshow(generated.reshape(28, 28), cmap=plt.cm.gray)
plt.axis('off');<jupyter_output><empty_output><jupyter_text>2D plot of the image classes in the latent spaceWe find again a similar organization of the latent space. Compared to the fully-connected VAE space, the differnt class labels seem slightly better separated. This could be a consequence of the slightly better fit we obtain from the convolutional models.<jupyter_code>x_test_encoded, _ = conv_encoder.predict(x_test_conv, batch_size=100)
plt.figure(figsize=(7, 6))
plt.scatter(x_test_encoded[:, 0], x_test_encoded[:, 1], c=y_test,
            cmap=plt.cm.tab10)
cb = plt.colorbar()
cb.set_ticks(list(id_to_labels.keys()))
cb.set_ticklabels(list(id_to_labels.values()))
cb.update_ticks()
plt.show()<jupyter_output><empty_output><jupyter_text>2D panel view of samples from the VAE manifoldThe following linearly spaced coordinates on the unit square were transformed through the inverse CDF (ppf) of the Gaussian to produce values of the latent variables z. This makes it possible to use a square arangement of panels that spans the gaussian prior of the latent space.<jupyter_code>n = 15  # figure with 15x15 panels
digit_size = 28
figure = np.zeros((digit_size * n, digit_size * n))
grid_x = norm.ppf(np.linspace(0.05, 0.95, n))
grid_y = norm.ppf(np.linspace(0.05, 0.95, n))

for i, yi in enumerate(grid_x):
    for j, xi in enumerate(grid_y):
        z_sample = np.array([[xi, yi]])
        x_decoded = conv_decoder.predict(z_sample)
        digit = x_decoded[0].reshape(digit_size, digit_size)
        figure[i * digit_size: (i + 1) * digit_size,
               j * digit_size: (j + 1) * digit_size] = digit

plt.figure(figsize=(10, 10))
plt.imshow(figure, cmap='Greys_r')
plt.show()<jupyter_output><empty_output><jupyter_text>Semi-supervised learningLet's reuse our encoder trained on many unlabeled samples to design a supervised model that can only use supervision from a small subset of samples with labels.To keep things simple we will just build a small supervised model on top of the latent representation defined by our encoder.We assume that we only have access to a small labeled subset with 50 examples per class (instead of 5000 examples per class in the full Fashion MNIST training set):<jupyter_code>rng = np.random.RandomState(42)

small_x_train = []
small_y_train = []
num_per_class = 50

for c in range(10):
    class_mask = np.where(y_train==c)[0]
    idx = rng.choice(class_mask, size=num_per_class, replace=False)
    small_x_train += [x_train_conv[idx]]
    small_y_train += [c] * num_per_class

small_x_train = np.vstack(small_x_train)
small_y_train = np.array(small_y_train)

# reshuffle our small dataset
perm = rng.permutation(range(small_y_train.shape[0]))
small_x_train = small_x_train[perm]
small_y_train = small_y_train[perm]

small_x_train.shape, small_y_train.shape<jupyter_output><empty_output><jupyter_text>**Exercise:**- Use `conv_encoder` to project `small_x_train` into the latent space;- Define a small supervised 10-class classification network and use `small_y_train` to train it;- What test accuracy can you reach? What is the chance level?- Suggest what could be changed to improve the quality of our classification on this small labeled dataset.<jupyter_code># TODO: implement me!
# define `small_x_train_encoded` for in the input training data
# define a model named `mdl` with its layers and its loss function.
# %load solutions/small_classifier.py
print(mdl.summary())
mdl.fit(small_x_train_encoded, small_y_train, 
        epochs=30, validation_data=[x_test_encoded, y_test])
from sklearn.metrics import confusion_matrix

y_pred = mdl.predict(x_test_encoded).argmax(axis=-1)
cnf_matrix = confusion_matrix(y_test, y_pred)
print(cnf_matrix)
import itertools

def plot_confusion_matrix(cm, classes, title='Confusion matrix',
                          cmap=plt.cm.Blues):
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], 'd'),
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')

class_names = [name for id, name in sorted(id_to_labels.items())]
plot_confusion_matrix(cnf_matrix, classes=class_names)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09694803409403355, 'fraction_numerical': 0.014737420951333517, 'mean_word_length': 4.6355748373101955, 'pattern_counts': {'":': 0, '<': 85, '<?xml version=': 0, '>': 86, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '179073', 'license': 'MIT', 'n_tokens_mistral': 6111, 'n_tokens_neox': 5701, 'n_words': 1681, 'path': 'labs/10_unsupervised_generative_models/Variational_AutoEncoders.ipynb', 'repo_name': 'Avinash439/lectures-labs', 'size': '18185'}