<jupyter_start><jupyter_text>Building the Sparse AutoencoderBuilding the sparse autoencoder is just as same as building the autoencoder except that here we use sparse regularizer in the encoder and decoder.   Import the librariesFirst, let us import the necessary libraries:<jupyter_code>import warnings
warnings.filterwarnings('ignore')

#modelling
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Dense, Conv2D, MaxPooling2D, UpSampling2D
from tensorflow.keras import backend as K
from tensorflow.keras import regularizers
import tensorflow as tf
tf.logging.set_verbosity(tf.logging.ERROR)

#plotting
import matplotlib.pyplot as plt
%matplotlib inline

#dataset
from keras.datasets import mnist
import numpy as np<jupyter_output>Using TensorFlow backend.<jupyter_text>Prepare the DatasetLet us load the MNIST dataset. We don't need the labels for autoencoder. Since we are reconstructing the given input we don't need the labels. So, we just load x_train for training and x_test for testing:<jupyter_code>(x_train, _), (x_test, _) = mnist.load_data()<jupyter_output><empty_output><jupyter_text>Normalize the data by dividing with max pixel value which is 255:<jupyter_code>x_train = x_train.astype('float32') / 255
x_test = x_test.astype('float32') / 255<jupyter_output><empty_output><jupyter_text>Reshape the images as 2D array:<jupyter_code>x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))
x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))<jupyter_output><empty_output><jupyter_text>Now the shape of data would become:<jupyter_code>print(x_train.shape, x_test.shape)<jupyter_output>((60000, 784), (10000, 784))<jupyter_text>Define the Sparse Regularizer We know that Sparse Regularizer is given as:$  \beta (\sum_{j=1}^{{l}^{(h)}} log \frac{\rho}{\hat{\rho_j}} + (1-\rho) log \frac{1-\rho}{1-\hat{\rho_j}} ) $ Define the sparse regularizer:<jupyter_code>def sparse_regularizer(activation_matrix):
    p = 0.01
    beta = 3
    p_hat = K.mean(activation_matrix) 
  
    KL_divergence = p*(K.log(p/p_hat)) + (1-p)*(K.log(1-p/1-p_hat))
    
    sum = K.sum(KL_divergence) 
   
    return beta * sum<jupyter_output><empty_output><jupyter_text>Define the EncoderDefine the encoder which takes the images as an input and returns the encodings.Define the size of the encodings:<jupyter_code>encoding_dim = 200<jupyter_output><empty_output><jupyter_text>Set the value of lambda:<jupyter_code>lambda_ = 0.001<jupyter_output><empty_output><jupyter_text>Define the shape of input:<jupyter_code>input_img = Input(shape=(784,))<jupyter_output><empty_output><jupyter_text>Define the encoder which takes the images as the inputs and returns the code:<jupyter_code>encoder = Dense(encoding_dim, 
                activation='sigmoid',
                kernel_regularizer=regularizers.l2(lambda_/2),activity_regularizer=sparse_regularizer)(input_img)<jupyter_output><empty_output><jupyter_text>Define the DecoderDefine the Decoder which takes the code returned by the encoder and returns the reconstructed input:<jupyter_code>decoder = Dense(784,
                activation='sigmoid',
                kernel_regularizer=regularizers.l2(lambda_/2),activity_regularizer=sparse_regularizer)(encoder)<jupyter_output><empty_output><jupyter_text>Build the modelNow that we defined encoder and decoder layers, we define the model which takes images as input and returns the output of the decoder layer which is the reconstructed image:<jupyter_code>model = Model(input_img, decoder)<jupyter_output><empty_output><jupyter_text>Let us look at summary of the model:<jupyter_code>model.summary()<jupyter_output>_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         (None, 784)               0         
_________________________________________________________________
dense (Dense)                (None, 200)               157000    
_________________________________________________________________
dense_1 (Dense)              (None, 784)               157584    
=================================================================
Total params: 314,584
Trainable params: 314,584
Non-trainable params: 0
_________________________________________________________________<jupyter_text>Compile the model with loss as binary cross entropy and we minimize the loss using sgd optimizer:<jupyter_code>model.compile(optimizer='sgd', loss='mse')<jupyter_output><empty_output><jupyter_text>Now, let us train the model.Generally, we feed the data to the model as model.fit(x,y) where x is the input and y is the label. But since autoencoders reconstruct its inputs, the input and output to the model should be the same. So we feed the data to the model as model.fit(x_train, x_train):<jupyter_code>model.fit(x_train, x_train, epochs=10, batch_size=256, shuffle=True, validation_data=(x_test, x_test))<jupyter_output>Train on 60000 samples, validate on 10000 samples
Epoch 1/10
60000/60000 [==============================] - 4s 67us/sample - loss: 0.5350 - val_loss: 0.5311
Epoch 2/10
60000/60000 [==============================] - 4s 58us/sample - loss: 0.5276 - val_loss: 0.5238
Epoch 3/10
60000/60000 [==============================] - 3s 56us/sample - loss: 0.5205 - val_loss: 0.5168
Epoch 4/10
60000/60000 [==============================] - 3s 55us/sample - loss: 0.5135 - val_loss: 0.5099
Epoch 5/10
60000/60000 [==============================] - 3s 57us/sample - loss: 0.5067 - val_loss: 0.5032
Epoch 6/10
60000/60000 [==============================] - 3s 55us/sample - loss: 0.5001 - val_loss: 0.4967
Epoch 7/10
60000/60000 [==============================] - 3s 55us/sample - loss: 0.4937 - val_loss: 0.4904
Epoch 8/10
60000/60000 [==============================] - 3s 55us/sample - loss: 0.4875 - val_loss: 0.4842
Epoch 9/10
60000/60000 [==============================] - 3s 56us/sample - loss: 0.4814 - val_[...]
{'fraction_non_alphanumeric': 0.15704809286898838, 'fraction_numerical': 0.05704809286898839, 'mean_word_length': 5.4091392136025505, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '227168', 'license': 'MIT', 'n_tokens_mistral': 2168, 'n_tokens_neox': 1857, 'n_words': 495, 'path': 'Chapter10/10.09 Building the Sparse Autoencoder.ipynb', 'repo_name': 'bharlow058/Packt-Deep-Learning', 'size': '6030'}