<jupyter_start><jupyter_text>CS109B Data Science 2: Advanced Topics in Data Science  Homework 5: Autoencoders **Harvard University****Spring 2020****Instructors:** <NAME>, <NAME>, <NAME><jupyter_code>#RUN THIS CELL 
import requests
from IPython.core.display import HTML
styles = requests.get("https://raw.githubusercontent.com/Harvard-IACS/2018-CS109A/master/content/styles/cs109.css").text
HTML(styles)
#RUN THIS CELL
import os
import pathlib
working_dir = pathlib.Path().absolute()
os.chdir(working_dir)<jupyter_output><empty_output><jupyter_text>INSTRUCTIONS- To submit your assignment follow the instructions given in Canvas.- This homework can be submitted in pairs.- Please restart the kernel and run the entire notebook again before you submit.<jupyter_code>import numpy as np
import seaborn as sns

from matplotlib import pyplot
import matplotlib.pylab as plt 
%matplotlib inline

from sklearn.decomposition import PCA
from sklearn.metrics import mean_squared_error

import tensorflow as tf
from tensorflow.keras import Input
from tensorflow.keras.models import Sequential, Model
from tensorflow.keras.layers import Dense, Dropout, Flatten, Activation, Reshape
from tensorflow.keras.layers import Conv2D, MaxPooling2D, UpSampling2D
from tensorflow.keras.optimizers import Adam, SGD<jupyter_output><empty_output><jupyter_text>Overview  In this homework, we will investigate autoencoders, how they are related to PCA (and in doing so, show that they can be a more powerful extension of this technique), and one possible application of autoencoders for outlier detection.  Question 1: Autoencoders and MNIST [50pts total] For this question, we will be using the [MNIST Dataset](https://en.wikipedia.org/wiki/MNIST_database) of handwritten digits, a simple standardized image dataset. The dataset consists of single-channel (black and white) 28x28 images, containing one digit each. We will see if it is feasible to encode (compress, in this case) the images into just 2 dimensions, a substantial compression ratio considering that the original vector has dimension 28x28=784.**1.1** [1pts] Load MNIST using `tf.keras.datasets.mnist.load_data()`, saving the training data as `x_train` and `y_train`, and the test data as `x_test`, `y_test`. Normalize the images to the range [0.,1.] by dividing by 255.**1.2** [1pts] Use `imshow` to show one image of your choice from the train set, specifying `cmap='gray'` to show the image in black and white.**1.3** [2pts] Construct and instance of `sklearn`'s `PCA` class, specifying that it only use the first 2 PCA components. Fit to `x_train` (Hint: you will need to use `reshape`), and project `x_train` down to its first 2 PCA components, saving the new array of shape (N,2) to `pca_latent_train`. This is the representation of all the images in `x_train` in a 2D latent space.**1.4** [2pts] Make a scatterplot of `pca_latent_train` with the point color designated by the corresponding class labels. Pick a reasonable color palette with enough of a contrast to clearly distinguish classes. **1.5** [8pts] Linear Autoencoder. Construct an encoder-decoder network with **linear activations** only, and **no biases**. The encoder and decoder should consist of one dense layer each, and the bottleneck dimension should be 2. The encoder and decoder should be their own separate models called `linear_encoder` and `linear_decoder`. Create the full linear autoencoder, call it `lae`, out of the encoder and decoder. Use a mean-squared-error reconstruction loss. Print the `summary()` for both the encoder and decoder networks, as well as the summary for the full linear autoencoder.**1.6** [4pts] Train your linear autoencoder `lae` on the train data, using `x_test` as validation data. Use enough epochs such that the training loss plateaus. Plot the train loss and validation (equivalent to test, in this case) loss as a function of epoch, in the same figure.**1.7** [3pts] Compute the `linear_encoder`'s latent space representation of `x_train`, calling the resulting array `lae_latent_train`. Create two scatterplots, side by side, using `subplots`, showing `pca_latent_train` (from **1.4**) and `lae_latent_train`, with points colored according to class label. Don't forget to title the two figures.**1.8** [3pts] What do you notice about the latent space representations in PCA and the LAE (linear autoencoder)? Does either one do a substantially better job at separating the 10 classes in the 2D latent space? --- *Bonus, but for no additional points: prove a relationship between the latent space representation in PCA and LAE for the same bottleneck dimension.***1.9** [3pts] What do you expect to happen if you added more dense layers (no biases) with only linear activations to your `linear_encoder` and `linear_decoder`? Would you expect a better reconstruction error?**1.10** [8pts] Construct a nonlinear (regular) autoencoder with at least 2 dense layers, with biases, in both the encoder and decoder parts. Call the encoder network `encoder` and the decoder network `decoder`, and the full autoencoder `ae`. Print the summaries for `encoder`, `decoder`, and `ae`.**1.11** [4pts] Train your autoencoder on `x_train`, using `x_test` as validation data. Train it for a reasonable number of epochs, using your best judgement on what that entails. As usual, plot the train loss and validation loss as a function of epoch.**1.12** [3pts] Compute the `encoder`'s latent space representation of `x_train`, calling the resulting array `ae_latent_train`. Plot the scatterplots of `pca_latent_train`, `lae_latent_train`, and `ae_latent_train` in a row using `subplots` so we can see them all simultaneously. What do you notice?**1.13** [6pts] Comparing reconstructed images. You will create arrays containing the reconstructed `x_test` using PCA, the linear autoencoder, and the regular autoencoder. For PCA, be sure to use the `pca` object you created and fit in **1.3** on the *train* data. You will project `x_test` onto its 2D latent space representation, and then convert it back, saving the result as `pca_recons_x_test`. For the linear autoencoder and the regular autoencoder, save the reconstructed `x_test` as `lae_recons_x_test` and `ae_recons_x_test` respectively. Now, you will create a 6 row by 4 column collection of subplots. Each row will correspond to an element of the test set (of your choice), with the columns being the PCA reconstruction, the LAE reconstruction, the AE reconstruction, and the original image. Be sure to title the subplots with 'PCA', 'LAE', 'AE', 'Original'. **1.14** [2pts] Finally, using `sklearn.metrics`'s `mean_squared_error`, report the average reconstruction error across the entire test set for PCA, LAE, and AE. Does the ordering agree with what you've seen in the previous questions? Does it support your conclusion in **1.8**?  Answers **1.1** [1pts] Load MNIST using `tf.keras.datasets.mnist.load_data()`, saving the training data as `x_train` and `y_train`, and the test data as `x_test`, `y_test`. Normalize the images to the range [0.,1.] by dividing by 255.<jupyter_code># your code here
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data(path='mnist.npz') # load MNIST data
x_train = x_train.astype('float32') / 255 # Normalize between 0 and 1
x_test = x_test.astype('float32') / 255 
x_test.shape<jupyter_output><empty_output><jupyter_text>**1.2** [1pts] Use `imshow` to show one image of your choice from the train set, specifying `cmap='gray'` to show the image in black and white.<jupyter_code># your code here
plt.imshow(x_train[0],cmap='gray')
plt.title('number: '+str(y_train[0]))<jupyter_output><empty_output><jupyter_text>**1.3** [2pts] Construct and instance of `sklearn`'s `PCA` class, specifying that it only use the first 2 PCA components. Fit to `x_train` (Hint: you will need to use `reshape`), and project `x_train` down to its first 2 PCA components, saving the new array of shape (N,2) to `pca_latent_train`. This is the representation of all the images in `x_train` in a 2D latent space.<jupyter_code># your code here
x_train = x_train.reshape((60000,-1))
pca = PCA(n_components=2)
fitted_pca = pca.fit(x_train)
pca_latent_train = fitted_pca.fit_transform(x_train)<jupyter_output><empty_output><jupyter_text>**1.4** [2pts] Make a scatterplot of `pca_latent_train` with the point color designated by the corresponding class labels. Pick a reasonable color palette with enough of a contrast to clearly distinguish classes.<jupyter_code># your code here
import pandas as pd
df = pd.DataFrame({'pca1':pca_latent_train[:,0],'pca2':pca_latent_train[:,1],'y':y_train})
marker_types = [".", "v", "1", "^", "s", "p", "P", "3", "H", "<", "|", "_", "x", "*"]
marker_colors = np.concatenate([np.array(plt.cm.tab10.colors),np.array(plt.cm.Pastel1.colors)])
for i in range(10):
    x1 = df[df.y==i].pca1
    x2 = df[df.y==i].pca2
    label = df[df.y ==i].y
    plt.scatter(x1,x2,c = marker_colors[label],s=10,label=i)
plt.xlabel("PC1 ({}%)".format(np.round(100*fitted_pca.explained_variance_ratio_[0],1)))
plt.ylabel("PC2 ({}%)".format(np.round(100*fitted_pca.explained_variance_ratio_[1],1)))
plt.legend()<jupyter_output><empty_output><jupyter_text>**1.5** [8pts] Linear Autoencoder. Construct an encoder-decoder network with **linear activations** only, and **no biases**. The encoder and decoder should consist of one dense layer each, and the bottleneck dimension should be 2. The encoder and decoder should be their own separate models called `linear_encoder` and `linear_decoder`. Create the full linear autoencoder, call it `lae`, out of the encoder and decoder. Use a mean-squared-error reconstruction loss. Print the `summary()` for both the encoder and decoder networks, as well as the summary for the full linear autoencoder.<jupyter_code># your code here
input_dim = x_train.shape[1]
encoding_dim = 2 
input_features = tf.keras.Input(shape=(input_dim,))
linear_encoder = tf.keras.layers.Dense(encoding_dim,name='encoder', activation=None)(input_features)
linear_decoder = tf.keras.layers.Dense(input_dim, name='decoder',activation=None)(linear_encoder)

#create autoencoder model which has input as input_features and outputs decoded
lae = tf.keras.models.Model(inputs = input_features,outputs = linear_decoder)

lae.compile(optimizer='adam', loss='mse')
print(lae.summary())<jupyter_output>Model: "model_17"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_12 (InputLayer)        [(None, 784)]             0         
_________________________________________________________________
encoder (Dense)              (None, 2)                 1570      
_________________________________________________________________
decoder (Dense)              (None, 784)               2352      
=================================================================
Total params: 3,922
Trainable params: 3,922
Non-trainable params: 0
_________________________________________________________________
None<jupyter_text>**1.6** [4pts] Train your linear autoencoder `lae` on the train data, using `x_test` as validation data. Use enough epochs such that the training loss plateaus. Plot the train loss and validation (equivalent to test, in this case) loss as a function of epoch, in the same figure.<jupyter_code># np.mean(x_train,axis =0).shape
# your code here
#Notice the y in .fit() below - This takes a minute to run (output suppressed)
x_test = x_test.reshape((10000,-1))
history = lae.fit(x_train, x_train,
                epochs=20,
                batch_size=64,
                shuffle=True,
                validation_data=(x_test,x_test),
                verbose = 1)

#plot our loss 
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model train vs validation loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper right')
plt.show()
# your code here<jupyter_output><empty_output><jupyter_text>**1.7** [3pts] Compute the `linear_encoder`'s latent space representation of `x_train`, calling the resulting array `lae_latent_train`. Create two scatterplots, side by side, using `subplots`, showing `pca_latent_train` (from **1.4**) and `lae_latent_train`, with points colored according to class label. Don't forget to title the two figures.<jupyter_code># your code here
encoder = tf.keras.Model(input_features, linear_encoder)
lae_latent_train = encoder.predict(x_train)

plt.figure(figsize=(12,6))
plt.subplot(121)
for i in range(10):
    x1 = df[df.y==i].pca1
    x2 = df[df.y==i].pca2
    label = df[df.y ==i].y
    plt.scatter(x1,x2,c = marker_colors[label],s=10,label=i)
plt.xlabel("PC1 ({}%)".format(np.round(100*fitted_pca.explained_variance_ratio_[0],1)))
plt.ylabel("PC2 ({}%)".format(np.round(100*fitted_pca.explained_variance_ratio_[1],1)))
plt.legend()
plt.title("PCA")

df_e = pd.DataFrame({'encode1':lae_latent_train [:,0],'encode2':lae_latent_train [:,1],'y':y_train})
plt.subplot(122)
for i in range(10):
    x1 = df_e[df_e.y==i].encode1
    x2 = df_e[df_e.y==i].encode2
    label = df_e[df_e.y ==i].y
    plt.scatter(x1,x2,c = marker_colors[label],s=10,label=i)
plt.xlabel("The first component")
plt.ylabel("The second components")
plt.legend()
plt.title("Autoencoder")
# your code here<jupyter_output><empty_output><jupyter_text>**1.8** [3pts] What do you notice about the latent space representations in PCA and the LAE (linear autoencoder)? Does either one do a substantially better job at separating the 10 classes in the 2D latent space? --- *Bonus, but for no additional points: prove a relationship between the latent space representation in PCA and LAE for the same bottleneck dimension.* *Your answer here*2-dimension PCA is a rotated and scaled version of the LAE.The relative position of classes is the same in two representations. They are as good as each other at separating these 10 classes in the 2D space. One can prove that PCA is a linear autoencoder with bias (due to decentralizing). **1.9** [3pts] What do you expect to happen if you added more dense layers (no biases) with only linear activations to your `linear_encoder` and `linear_decoder`? Would you expect a better reconstruction error? *Your answer here*The performance won't be better. We'll get the same linear autoencoder because the multiplication of two linear layers is still linear. Adding more dense layer without introducing nonlinearity doesn't make sense. The reconstruction error will also be the same. **1.10** [8pts] Construct a nonlinear (regular) autoencoder with at least 2 dense layers, with biases, in both the encoder and decoder parts. Call the encoder network `encoder` and the decoder network `decoder`, and the full autoencoder `ae`. Print the summaries for `encoder`, `decoder`, and `ae`.<jupyter_code># your code here
# your code here
input_dim = x_train.shape[1]
encoding_dim = 2 
input_features = tf.keras.Input(shape=(input_dim,))
encoder_1 = tf.keras.layers.Dense(input_dim,name='encoder1', activation='relu',use_bias=True)(input_features)
encoder_2 = tf.keras.layers.Dense(encoding_dim,name='encoder2', activation='relu',use_bias=True)(encoder_1)
# encoder = tf.keras.models.Model(inputs = input_features,outputs = encoder_2)
encoded_features = tf.keras.Input(shape=(encoding_dim,))

decoder_1 = tf.keras.layers.Dense(input_dim, name='decoder1', activation='relu',use_bias=True)(encoder_2)
decoder_2 = tf.keras.layers.Dense(input_dim, name='decoder2', activation='relu',use_bias=True)(decoder_1)


decoder_3 = tf.keras.layers.Dense(input_dim, name='decoder3', activation='relu',use_bias=True)(encoded_features)
decoder_4 = tf.keras.layers.Dense(input_dim, name='decoder4', activation='relu',use_bias=True)(decoder_3)
decoder = tf.keras.models.Model(inputs = encoded_features,outputs = decoder_4)

# #create autoencoder model which has input as input_features and outputs decoded
ae = tf.keras.models.Model(inputs = input_features,outputs = decoder_2)

# ae.compile(optimizer='adam', loss='mse')
# print(decoder.summary())
print(ae.summary())<jupyter_output>Model: "model_10"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_13 (InputLayer)        [(None, 784)]             0         
_________________________________________________________________
encoder1 (Dense)             (None, 784)               615440    
_________________________________________________________________
encoder2 (Dense)             (None, 2)                 1570      
_________________________________________________________________
decoder1 (Dense)             (None, 784)               2352      
_________________________________________________________________
decoder2 (Dense)             (None, 784)               615440    
=================================================================
Total params: 1,234,802
Trainable params: 1,234,802
Non-trainable params: 0
________________________________________________[...]<jupyter_text>**1.11** [4pts] Train your autoencoder on `x_train`, using `x_test` as validation data. Train it for a reasonable number of epochs, using your best judgement on what that entails. As usual, plot the train loss and validation loss as a function of epoch.<jupyter_code># your code here
ae.compile(optimizer='adam', loss='mse')
x_test = x_test.reshape((10000,-1))
history_ae = ae.fit(x_train, x_train,
                epochs=20,
                batch_size=64,
                shuffle=True,
                validation_data=(x_test,x_test),
                verbose = 1)

#plot our loss 
plt.plot(history_ae.history['loss'])
plt.plot(history_ae.history['val_loss'])
plt.title('model train vs validation loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper right')
plt.show()
# your code here<jupyter_output><empty_output><jupyter_text>**1.12** [3pts] Compute the `encoder`'s latent space representation of `x_train`, calling the resulting array `ae_latent_train`. Plot the scatterplots of `pca_latent_train`, `lae_latent_train`, and `ae_latent_train` in a row using `subplots` so we can see them all simultaneously. What do you notice?<jupyter_code># your code here
ae_latent_train = encoder.predict(x_train)
# your code here
# your code here

plt.figure(figsize=(12,6))
plt.subplot(131)
for i in range(10):
    x1 = df[df.y==i].pca1
    x2 = df[df.y==i].pca2
    label = df[df.y ==i].y
    plt.scatter(x1,x2,c = marker_colors[label],s=10,label=i)
plt.xlabel("PC1 ({}%)".format(np.round(100*fitted_pca.explained_variance_ratio_[0],1)))
plt.ylabel("PC2 ({}%)".format(np.round(100*fitted_pca.explained_variance_ratio_[1],1)))
plt.legend()
plt.title("PCA")

df_lae = pd.DataFrame({'encode1':lae_latent_train [:,0],'encode2':lae_latent_train [:,1],'y':y_train})
plt.subplot(132)
for i in range(10):
    x1 = df_lae[df_lae.y==i].encode1
    x2 = df_lae[df_lae.y==i].encode2
    label = df_lae[df_lae.y ==i].y
    plt.scatter(x1,x2,c = marker_colors[label],s=10,label=i)
plt.xlabel("The first component")
plt.ylabel("The second components")
plt.legend()
plt.title("Linear Autoencoder")

df_ae = pd.DataFrame({'encode1':ae_latent_train [:,0],'encode2':ae_latent_train [:,1],'y':y_train})
plt.subplot(133)
for i in range(10):
    x1 = df_ae[df_ae.y==i].encode1
    x2 = df_ae[df_ae.y==i].encode2
    label = df_ae[df_ae.y ==i].y
    plt.scatter(x1,x2,c = marker_colors[label],s=10,label=i)
plt.xlabel("The first component")
plt.ylabel("The second components")
plt.legend()
plt.title("Non-linear Autoencoder")<jupyter_output><empty_output><jupyter_text>*Your answer here*The non-linear perform better on separating different classes. The validation loss of the non-linear model is 0.042, less than that of the linear model 0.057. In the encoded layer, spots of different classes radiate from the original point in different directions and distances. In the PCA or linear autoencoder, classes like 5 and 6 almost overlap with other classes and the spots are very diffused. However, in the nonlinear autoencoder, every class has its only occupied region that's not overlapping totally with another class. **1.13** [6pts] Comparing reconstructed images. You will create arrays containing the reconstructed `x_test` using PCA, the linear autoencoder, and the regular autoencoder. For PCA, be sure to use the `pca` object you created and fit in **1.3** on the *train* data. You will project `x_test` onto its 2D latent space representation, and then convert it back, saving the result as `pca_recons_x_test`. For the linear autoencoder and the regular autoencoder, save the reconstructed `x_test` as `lae_recons_x_test` and `ae_recons_x_test` respectively. Now, you will create a 6 row by 4 column collection of subplots. Each row will correspond to an element of the test set (of your choice), with the columns being the PCA reconstruction, the LAE reconstruction, the AE reconstruction, and the original image. Be sure to title the subplots with 'PCA', 'LAE', 'AE', 'Original'.<jupyter_code># your code here
pca = PCA(n_components=2)
pca_test = pca.fit_transform(x_test)
pca_recons_x_test = pca.inverse_transform(pca_test).reshape(10000,28,28)

ae_recons_x_test = ae.predict(x_test).reshape(10000,28,28)
lae_recons_x_test = lae.predict(x_test).reshape(10000,28,28)
# your code here
plt.figure(figsize=(12,18))
for i in range(6):
    ax1 = plt.subplot2grid((6,4), (i,0))
    ax1.imshow(pca_recons_x_test[i])
    ax1.set_title("PCA")
    ax2 = plt.subplot2grid((6,4), (i,1))
    ax2.imshow(lae_recons_x_test[i])
    ax2.set_title("Linear Autoencoder")
    ax3 = plt.subplot2grid((6,4), (i,2))
    ax3.imshow(ae_recons_x_test[i])
    ax3.set_title("Autoencoder")
    ax4 = plt.subplot2grid((6,4), (i,3))
    ax4.imshow(x_test[i].reshape(28,28))
    ax4.set_title("Original")<jupyter_output><empty_output><jupyter_text>**1.14** [2pts] Finally, using `sklearn.metrics`'s `mean_squared_error`, report the average reconstruction error across the entire test set for PCA, LAE, and AE. Does the ordering agree with what you've seen in the previous questions? Does it support your conclusion in **1.8**?<jupyter_code># your code here
x_test = x_test.reshape(10000,-1)
pca_mse = mean_squared_error(x_test,pca_recons_x_test.reshape(10000,-1))
lae_mse = mean_squared_error(x_test,lae_recons_x_test.reshape(10000,-1))
ae_mse = mean_squared_error(x_test,ae_recons_x_test.reshape(10000,-1))
print("PCA: ",pca_mse,";LAE: ",lae_mse,";AE: ",ae_mse)<jupyter_output>PCA:  0.055549856 ;LAE:  0.055732183 ;AE:  0.04223134<jupyter_text>*Your answer here*The reconstruction error of AE is the minimum among the three method. The mse is similar using PCA and LAE. Since the ability to separate classes in the latent encoded layer is directly related to the accuracy of the reconstruction, this result supports the suggestion in 1.8 that the PCA and LAE's performances are similar in terms of separating different classes   Question 2: Convolutional Autoencoders and Outlier Detection [50pts total] For this question, we will be using a modified version of a subset of MNIST. We have hidden some images of handwritten letters in the dataset `data/cs109b-mnist-mix.csv` amongst thousands of handwritten digits. The dataset is provided as a csv, where each row is an image, and each column gives the value of a given pixel in a flattened 28 by 28 image. It would be very tedious to have humans flip through every image to find the letters, so instead we will exploit a neat feature of autoencoders, outlier detection. This method turns a disadvantage of autoencoders, namely, their inability to properly reconstruct data very dissimilar to what they were trained on, into an advantage. You will also be constructing a convolutional autoencoder, which tends to work a lot better for reconstructing images, all while using substantially fewer parameters.**2.1** [2pts] Load and normalize (by dividing by 255) the modified dataset from `data/cs109b-mnist-mix.csv`, and reshape it to (-1, 28, 28), saving the array as `x_cs109b`. Using `imshow` and `cmap='gray'`, plot one image of your choice from this dataset.**2.2** [15pts] Create a convolutional autoencoder called `cae`. This time you don't need separate references to the encoder and decoder parts since we only intend to use the full autoencoder. You may use a combination of `Conv2D`, `MaxPooling2D`, `Flatten`, `Dense`, `Reshape`, and `UpSampling2D` layers. You may use any number of these layers, and if you are unfamiliar with any of these layers we encourage you to look at their TF Keras documentation. You will have to experiment with an appropriate bottleneck size to complete the rest of question **2**. As always, print the `summary()` of your model.**2.3** [10pts] Train your convolutional autoencoder on `x_train` (from **problem 1**, MNIST), using `x_test` as validation data. Plot the train/validation loss versus epoch. This will adapt the convolutional autoencoder to the type of data we expect, handwritten digits.**2.4** [4pts] Pass `x_test` through your convolutional autoencoder (CAE), calling the resulting reconstruction of the dataset `cae_recons_x_test`. To see how well your CAE is performing, we will visually inspect some of its reconstructions. Make an array of subplots of 6 rows and 2 columns, with the rows being different elements of the test set (your choice) and the columns being the CAE reconstruction and the Original image. How good is the reconstruction? How does it compare to the reconstruction of your dense autoencoder from **1.13**?**2.5** [4pts] Construct an instance of `tf.keras.losses.MeanSquaredError(reduction=tf.keras.losses.Reduction.NONE)` and use it to calculate the MSE reconstruction error of every element in `x_test`. Save this array as `mse_x_test`; you want this output to be a 1D array so consider the required shapes of what you feed in to the `MeanSquaredError` object. We will now pass the mystery dataset through the CAE: reconstruct `x_cs109b`, saving the result as `cae_recons_x_cs109b`. Compute the reconstruction errors, saving the result as `mse_x_cs109b`.**2.6** [5pts] Using subplots, plot the histograms of `mse_x_test` and `mse_x_cs109b` side-by-side. For the most part, do they look like they are coming from similar types of data? Pick a reasonable threshold value for reconstruction error based on the histogram for `mse_x_cs109b`; plot this threshold as a vertical line on the histogram. Be sure to explain your choice of threshold. Beyond this threshold, you will examine the images to see if you can find letters. Print how many images lie beyond this threshold.**2.7** [10pts] Use the subset of data determined by your threshold to find the letters in `x_cs109b`, displaying them as images. Show your work! Create a 2-column table indicating the letter (e.g., 'a') and the index where it is located (e.g. '9728'). There are a few letters total. To get full credit you need to find most of them (you should not need to examine more than a few tens of images - if so, consider a different threshold in **2.6** or check your CAE performance).You will have been able to find the majority of the letters hidden in the dataset of thousands of images, while only having to manually look at 1/100th of the dataset. This demonstrates how autoencoders could be used as a cheap preprocessing step to draw out the most "interesting" data.  Answers **2.1** [2pts] Load and normalize (by dividing by 255) the modified dataset from `data/cs109b-mnist-mix.csv`, and reshape it to (-1, 28, 28), saving the array as `x_cs109b`. Using `imshow` and `cmap='gray'`, plot one image of your choice from this dataset.<jupyter_code># your code here
x_cs109b = np.genfromtxt('cs109b-mnist-mix.csv', delimiter=',')
x_cs109b = x_cs109b.astype('float32') / 255 
# your code here
x_cs109b = x_cs109b.reshape(-1,28,28,1)
x_cs109b.shape
plt.imshow(x_cs109b[0].reshape(28,28),cmap='gray')<jupyter_output><empty_output><jupyter_text>**2.2** [15pts] Create a convolutional autoencoder called `cae`. This time you don't need separate references to the encoder and decoder parts since we only intend to use the full autoencoder. You may use a combination of `Conv2D`, `MaxPooling2D`, `Flatten`, `Dense`, `Reshape`, and `UpSampling2D` layers. You may use any number of these layers, and if you are unfamiliar with any of these layers we encourage you to look at their TF Keras documentation. You will have to experiment with an appropriate bottleneck size to complete the rest of question **2**. As always, print the `summary()` of your model.<jupyter_code># your code here

cae = tf.keras.models.Sequential() 
encoding_dim = 12
# Encoder
cae.add(tf.keras.layers.Conv2D(32, (3, 3), name = 'Con2D1_encode',activation='relu', padding='same',input_shape=(28,28,1)))
cae.add(tf.keras.layers.MaxPool2D(2,2,name = 'MaxPool2D1_encode')) ##14*14
cae.add(tf.keras.layers.Conv2D(32, (3, 3), name = 'Con2D2_encode',activation='relu', padding='same',input_shape=(28,28,1)))
cae.add(tf.keras.layers.MaxPool2D(2,2,name = 'MaxPool2D2_encode')) ##7*7
cae.add(tf.keras.layers.Flatten(name='Flatten'))  ## 49
cae.add(tf.keras.layers.Dense(encoding_dim,name='Dense_encode', activation='relu',use_bias=True))
# Decoder
cae.add(tf.keras.layers.Dense(49,name='Dense_decode', activation='relu',use_bias=True))
cae.add(tf.keras.layers.Reshape((7,7,1),name='Reshape_decode'))
cae.add( tf.keras.layers.Conv2D(32, (3, 3),name = 'Con2D1_decode', activation='relu', padding='same')) 
cae.add( tf.keras.layers.UpSampling2D((2, 2),name = 'UpSampling2D1_encode') ) ## 14*14
cae.add( tf.keras.layers.Conv2D(32, (3, 3),name = 'Con2D2_decode', activation='relu', padding='same')) 
cae.add( tf.keras.layers.UpSampling2D((2, 2),name = 'UpSampling2D2_encode' ))  ## 28:28
cae.add( tf.keras.layers.Conv2D(1, (1, 1),name = 'Con2D3_decode', activation='sigmoid', padding='same')) 
cae.compile(optimizer='adam', loss='mse')
print(cae.summary())<jupyter_output>Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
Con2D1_encode (Conv2D)       (None, 28, 28, 32)        320       
_________________________________________________________________
MaxPool2D1_encode (MaxPoolin (None, 14, 14, 32)        0         
_________________________________________________________________
Con2D2_encode (Conv2D)       (None, 14, 14, 32)        9248      
_________________________________________________________________
MaxPool2D2_encode (MaxPoolin (None, 7, 7, 32)          0         
_________________________________________________________________
Flatten (Flatten)            (None, 1568)              0         
_________________________________________________________________
Dense_encode (Dense)         (None, 12)                18828     
________________________________________________________[...]<jupyter_text>**2.3** [10pts] Train your convolutional autoencoder on `x_train` (from **problem 1**, MNIST), using `x_test` as validation data. Plot the train/validation loss versus epoch. This will adapt the convolutional autoencoder to the type of data we expect, handwritten digits.<jupyter_code>%time 
checkpoint_path_celeb ='mnist/cp.ckpt'
checkpoint_dir = os.path.dirname(checkpoint_path_celeb)
# Save Codes
ae_callback = tf.keras.callbacks.ModelCheckpoint(filepath = checkpoint_path_celeb,
                                                     save_weights_only = True,
                                                     verbose=1)
x_test = x_test.reshape((-1,28,28,1))
x_train = x_train.reshape((-1,28,28,1))
# your code here

# cae.load_weights('cae_model.h5')
history_cae = cae.fit(x_train, x_train,
                epochs=3,
                batch_size=64,
                shuffle=True,
                validation_data=(x_test,x_test),
                verbose = 1,
                callbacks = [ae_callback])
#plot our loss 
plt.plot(history_cae.history['loss'])
plt.plot(history_cae.history['val_loss'])
plt.title('model train vs validation loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper right')
plt.show()
# your code here
cae.save('cae_model.h5')<jupyter_output><empty_output><jupyter_text>**2.4** [4pts] Pass `x_test` through your convolutional autoencoder (CAE), calling the resulting reconstruction of the dataset `cae_recons_x_test`. To see how well your CAE is performing, we will visually inspect some of its reconstructions. Make an array of subplots of 6 rows and 2 columns, with the rows being different elements of the test set (your choice) and the columns being the CAE reconstruction and the Original image. How good is the reconstruction? How does it compare to the reconstruction of your dense autoencoder from **1.13**?<jupyter_code># your code here
cae.load_weights('cae_model.h5')
cae_recons_x_test = cae.predict(x_test)
# your code here
plt.figure(figsize=(6,18))
for i in range(6):
    ax1 = plt.subplot2grid((6,2), (i,0))
    ax1.imshow(cae_recons_x_test[i].reshape(28,28))
    ax1.set_title("CAE")
    ax2 = plt.subplot2grid((6,2), (i,1))
    ax2.imshow(x_test[i].reshape(28,28))
    ax2.set_title("Original")<jupyter_output><empty_output><jupyter_text>*Your answer here* **2.5** [4pts] Construct an instance of `tf.keras.losses.MeanSquaredError(reduction=tf.keras.losses.Reduction.NONE)` and use it to calculate the MSE reconstruction error of every element in `x_test`. Save this array as `mse_x_test`; you want this output to be a 1D array so consider the required shapes of what you feed in to the `MeanSquaredError` object. We will now pass the mystery dataset through the CAE: reconstruct `x_cs109b`, saving the result as `cae_recons_x_cs109b`. Compute the reconstruction errors, saving the result as `mse_x_cs109b`.<jupyter_code># your code here
mse = tf.keras.losses.MeanSquaredError(reduction=tf.keras.losses.Reduction.NONE)
mse_x_test=mse(x_test.reshape(-1,28*28),cae_recons_x_test.reshape(-1,28*28)).numpy()
mse_x_test
# your code here
cae_recons_x_cs109b = cae.predict(x_cs109b)
mse_x_cs109b = mse(x_cs109b.reshape(-1,28*28),cae_recons_x_cs109b.reshape(-1,28*28)).numpy()
mse_x_cs109b<jupyter_output><empty_output><jupyter_text>**2.6** [5pts] Using subplots, plot the histograms of `mse_x_test` and `mse_x_cs109b` side-by-side. For the most part, do they look like they are coming from similar types of data? Pick a reasonable threshold value for reconstruction error based on the histogram for `mse_x_cs109b`; plot this threshold as a vertical line on the histogram. Be sure to explain your choice of threshold. Beyond this threshold, you will examine the images to see if you can find letters. Print how many images lie beyond this threshold.<jupyter_code># your code here
plt.subplot(121)
plt.hist(mse_x_test,bins=50)
plt.title("X_test")
plt.xlabel("MSE")
plt.subplot(122)
plt.hist(mse_x_cs109b,bins=50)
thresh = 0.085
plt.axvline(thresh,c='r',ls='--')
plt.title("X_cs109b")
plt.xlabel("MSE")
print("There are ",sum(mse_x_cs109b>thresh)," images beyong the threshold ", thresh)
# your code here<jupyter_output><empty_output><jupyter_text>*Your answer here* **2.7** [10pts] Use the subset of data determined by your threshold to find the letters in `x_cs109b`, displaying them as images. Show your work! Create a 2-column table indicating the letter (e.g., 'a') and the index where it is located (e.g. '9728'). There are a few letters total. To get full credit you need to find most of them (you should not need to examine more than a few tens of images - if so, consider a different threshold in **2.6** or check your CAE performance).You will have been able to find the majority of the letters hidden in the dataset of thousands of images, while only having to manually look at 1/100th of the dataset. This demonstrates how autoencoders could be used as a cheap preprocessing step to draw out the most "interesting" data.<jupyter_code># your code here
idx = 0
plt.figure(figsize=(30,10))
letters_img = []
idxs = []
for i in range(x_cs109b.shape[0]):
    if mse_x_cs109b[i]>thresh:
        row = np.floor(idx/10)
        col = idx-row*10
        ax = plt.subplot2grid((3,10), (int(row),int(col)))
        ax.imshow(x_cs109b[i].reshape(28,28),cmap='gray')
        idx +=1
        if idx in [7,8,13,24,25]:
            letters_img.append(x_cs109b[i].reshape(28,28))
        idxs.append(i)
letters = ["Q","J","e","f","M"]
plt.figure(figsize=(15,4))
for i,letter in enumerate(letters):
    order = int("15"+str(i+1))
    plt.subplot(order)
    plt.imshow(letters_img[i],cmap='gray')
    plt.title(letter+"; ""index:"+str(idxs[i]))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11353699932111337, 'fraction_numerical': 0.029979633401221995, 'mean_word_length': 5.1091572660915725, 'pattern_counts': {'":': 0, '<': 85, '<?xml version=': 0, '>': 86, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '418075', 'license': 'MIT', 'n_tokens_mistral': 12114, 'n_tokens_neox': 11167, 'n_words': 3958, 'path': 'content/HW/hw5/cs109b_hw5_students_v1.ipynb', 'repo_name': 'jialululu/2020-CS109B', 'size': '36849'}