<jupyter_start><jupyter_text>Autoencoder![](https://blog.keras.io/img/ae/autoencoder_schema.jpg) This tutorial is based on the official Autoencoder tutorial on the Keras blog created by <NAME>.   What is an Autoencoder?An autoencoder is a type of artificial neural network which has the goal of learning the identity function. In other words it tries to copy its input to its output. But why would we need this?If the only purpose of an autoencoder would be to try and copy the input it would be useless. But by placing constraints on the network we can discover interesting structures in the data. What can you use it for?We can use an autoencoder for data denoising, dimensionality reduction as well as generating new data that is simular to the data we trained it on.  Autoencoder in KerasAutoencoders can be build with multiple different neural network architectures like simple FFNN as well as CNN and RNN and Keras let's use implement all of them.  Imports<jupyter_code>import numpy as np
import matplotlib.pyplot as plt

from keras.layers import Input, Dense, Conv2D, MaxPooling2D, UpSampling2D, LSTM 
from keras.models import Model
from keras.datasets import mnist<jupyter_output>Using TensorFlow backend.<jupyter_text>Loading in dataset<jupyter_code>(x_train, y_train), (x_test, y_test) = mnist.load_data()
del y_train
del y_test

# scale data between 0 and 1
x_train = x_train.astype('float32') / 256
x_test = x_test.astype('float32') / 256

# Reshape data
x_train = x_train.reshape(len(x_train), 28*28)
x_test = x_test.reshape(len(x_test), 28*28)

print('Training data shape:', x_train.shape)
print('Testing data shape:', x_test.shape)<jupyter_output>Training data shape: (60000, 784)
Testing data shape: (10000, 784)<jupyter_text>Simple autoencoder Let's start simple by creating an really simple autoencoder which tries to learn the identity function of the MNIST dataset.<jupyter_code>input_layer = Input(shape=(784,))
# encoded layer. compresses data by the factor 12.25 (784/64=12.25)
encoded = Dense(64, activation='relu')(input_layer)
# decoded layer. reconstructs the input (units are equal to the input dimension of the image)
decoded = Dense(784, activation='sigmoid')(encoded)<jupyter_output><empty_output><jupyter_text>Creating models:<jupyter_code># this model maps an input to its reconstruction
autoencoder = Model(input_layer, decoded)
# this model maps an input to its encoded representation
encoder = Model(input_layer, encoded)
# for the decoder we need to create an input layer which has the dimensionality of the encoded layer
input_layer_dec = Input(shape=(64,))
# retrieve the last layer of the autoencoder model and place it after the input layer
decoder_layer = autoencoder.layers[-1](input_layer_dec)

decoder = Model(input_layer_dec, decoder_layer)<jupyter_output><empty_output><jupyter_text>Configure and fit our autoencoder model:<jupyter_code>autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')

autoencoder.fit(x_train, x_train, epochs=50,
                batch_size=256, shuffle=True,
                validation_data=(x_test, x_test))
encoded_images = encoder.predict(x_test)
decoded_images = decoder.predict(encoded_images) # takes the output from the encoder as input<jupyter_output><empty_output><jupyter_text>Visualize results using matplotlib:<jupyter_code>def visualize_data(data, size):
    n = 10 # number of digits that will be displayed
    plt.figure(figsize=(20, 4))
    plt.gray()
    for i in range(n):
        ax = plt.subplot(2, n, i+1)
        plt.imshow(data[i].reshape(size, size)) # reshape the image back to its normal shape
        # disable axis
        ax.get_xaxis().set_visible(False)
        ax.get_yaxis().set_visible(False)
    plt.show()    
visualize_data(x_test, 28)
visualize_data(decoded_images, 28)<jupyter_output><empty_output><jupyter_text>We can also display our encoded images so we can see what shapes the algorithm thinks are important.<jupyter_code>visualize_data(encoded_images, 8)<jupyter_output><empty_output><jupyter_text>Deep autoencoderTo get better results we can use more layers or we could add sparsity using the activity_regularizer parameter.<jupyter_code>input_layer = Input(shape=(784,))
encoded = Dense(128, activation='relu')(input_layer)
encoded = Dense(64, activation='relu')(encoded)

decoded = Dense(128, activation='relu')(encoded)
decoded = Dense(784, activation='sigmoid')(decoded)
autoencoder = Model(input_layer, decoded)
autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')

autoencoder.fit(x_train, x_train, epochs=100,
                batch_size=256, shuffle=True,
                validation_data=(x_test, x_test))
autoencoded_images = autoencoder.predict(x_test)
visualize_data(x_test, 28)
visualize_data(autoencoded_images, 28)<jupyter_output><empty_output><jupyter_text>Convolutional autoencoderSince our inputs are images, it makes sense to use convolutional neural networks (CNNs) since they have shown to achieve state of the art performance on image data.We will use MaxPolling to down-sample our data in the encoder and we will use UpSampling for the decoder.<jupyter_code># Reshape data for Convolutional Layer
x_train = x_train.reshape(len(x_train), 28, 28, 1)
x_test = x_test.reshape(len(x_test), 28, 28, 1)
print('Training data shape:', x_train.shape)
print('Testing data shape:', x_test.shape)
input_layer = Input(shape=(28, 28, 1))

x = Conv2D(16, (3, 3), activation='relu', padding='same')(input_layer)
x = MaxPooling2D((2, 2), padding='same')(x)
x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)
x = MaxPooling2D((2, 2), padding='same')(x)
x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)
encoded = MaxPooling2D((2,2), padding='same')(x)

# encoded representation is (4, 4, 8) i.e. 128-dimensional 

x = Conv2D(8, (3, 3), activation='relu', padding='same')(encoded)
x = UpSampling2D((2, 2))(x)
x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)
x = UpSampling2D((2, 2))(x)
x = Conv2D(16, (3, 3), activation='relu')(x)
x = UpSampling2D((2, 2))(x)
decoded = Conv2D(1, (3, 3), activation='sigmoid', padding='same')(x)
autoencoder = Model(input_layer, decoded)
autoencoder.compile(loss='binary_crossentropy', optimizer='adadelta')
autoencoder.fit(x_train, x_train, epochs=50, 
                batch_size=128, shuffle=True,
                validation_data=(x_test, x_test))
autoencoded_images = autoencoder.predict(x_test)
visualize_data(x_test, 28)
visualize_data(autoencoded_images, 28)<jupyter_output><empty_output><jupyter_text>Image denoisingWe can denoise our image by letting our autoencoder map noisy images to clean once.  Creating noisy data<jupyter_code>noise_factor = 0.5
x_train_noisy = x_train + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=x_train.shape) 
x_test_noisy = x_test + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=x_test.shape) 

x_train_noisy = np.clip(x_train_noisy, 0., 1.)
x_test_noisy = np.clip(x_test_noisy, 0., 1.)
visualize_data(x_test_noisy, 28)<jupyter_output><empty_output><jupyter_text>In order to improve the quality of the image we will need more filters for our Convolutional Layers.<jupyter_code>input_layer = Input(shape=(28, 28, 1))

x = Conv2D(32, (3, 3), activation='relu', padding='same')(input_layer)
x = MaxPooling2D((2, 2), padding='same')(x)
x = Conv2D(32, (3, 3), activation='relu', padding='same')(x)
encoded = MaxPooling2D((2, 2), padding='same')(x)

# at this point the representation is (7, 7, 32)

x = Conv2D(32, (3, 3), activation='relu', padding='same')(encoded)
x = UpSampling2D((2, 2))(x)
x = Conv2D(32, (3, 3), activation='relu', padding='same')(x)
x = UpSampling2D((2, 2))(x)
decoded = Conv2D(1, (3, 3), activation='sigmoid', padding='same')(x)
autoencoder = Model(input_layer, decoded)
autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')
autoencoder.fit(x_train_noisy, x_train, epochs=100, 
                batch_size=128, shuffle=True,
                validation_data=(x_test_noisy, x_test))
denoised_data = autoencoder.predict(x_train_noisy)

visualize_data(x_train_noisy, 28)
visualize_data(denoised_data, 28)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11541297935103245, 'fraction_numerical': 0.031096361848574238, 'mean_word_length': 6.045021645021645, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '477778', 'license': 'MIT', 'n_tokens_mistral': 2941, 'n_tokens_neox': 2709, 'n_words': 767, 'path': 'Keras-Tutorials/7. Autoencoder/Autoencoder.ipynb', 'repo_name': 'eratiniou/Tutorials', 'size': '8146'}