<jupyter_start><jupyter_text>Application to image denoisingWe will train the autoencoder to map noisy images to clean digits images.You are provided two sets of images, train and test. These images contain various styles of text, to which synthetic noise has been added to simulate real-world, messy artifacts. The training set includes the test without the noise (train_cleaned). You must create an algorithm to clean the images in the test set.  Load Keras, TensorFlow, and the training/testing datasets<jupyter_code>import os
import sys
import numpy as np

module_path = os.path.abspath(os.path.join('..'))
if module_path not in sys.path:
    sys.path.append(module_path)

import matplotlib.pyplot as plt
%matplotlib inline
plt.rcParams['figure.figsize'] = (8,8) # Make the figures a bit bigger

from keras.layers import Input, Conv2D, MaxPooling2D, UpSampling2D
from keras.models import Model

from utils.helper import load_document_images_cropped
# Seed for reproducibility
np.random.seed(42)<jupyter_output><empty_output><jupyter_text>Load training data<jupyter_code># the data, shuffled and split between train and test sets
img_size = 64
img_row, img_col = img_size, img_size
(X_train, y_train), (X_test, _) = load_document_images_cropped(img_size)

print("X_train original shape", X_train.shape)
print("y_train original shape", y_train.shape)
print("X_test original shape", X_test.shape)
img_nbr = 143
plt.figure(figsize=(20, 10))

ax = plt.subplot(1, 2, 1)
plt.imshow(X_train[img_nbr].reshape(img_row, img_col))
plt.gray()
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)


ax = plt.subplot(1, 2, 2)
plt.imshow(y_train[img_nbr].reshape(img_row, img_col))
plt.gray()
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)

plt.show()<jupyter_output><empty_output><jupyter_text>Format the data for training<jupyter_code>X_train = X_train.astype('float32')
y_train = y_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
y_train /= 255
X_test /= 255

X_train = np.reshape(X_train, (len(X_train), img_row, img_col, 1))
y_train = np.reshape(y_train, (len(y_train), img_row, img_col, 1))
X_test = np.reshape(X_test, (len(X_test), img_row, img_col, 1))
print("Training matrix shape", X_train.shape)
print("Testing matrix shape", X_test.shape)<jupyter_output><empty_output><jupyter_text>Define the Model<jupyter_code>input_img = Input(shape=(img_row, img_col, 1))

## Your model here

model.compile(loss='binary_crossentropy', optimizer='adadelta')
# prints a summary representation of the model
# model.summary()<jupyter_output><empty_output><jupyter_text>Train the model<jupyter_code>model.fit(X_train, y_train, epochs=1, batch_size=1, shuffle=True, verbose=2)<jupyter_output><empty_output><jupyter_text>Visualize the resultNow let's take a look at the results. Top, the noisy images fed to the network, and bottom, the images are reconstructed by the network.<jupyter_code>test_denoised = model.predict(X_test)
img_nbr = 15
plt.figure(figsize=(20, 10))

ax = plt.subplot(1, 2, 1)
plt.imshow(X_test[img_nbr].reshape(img_row, img_col))
plt.gray()
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)


ax = plt.subplot(1, 2, 2)
plt.imshow(test_denoised[img_nbr].reshape(img_row, img_col))
plt.gray()
ax.get_xaxis().set_visible(False)
ax.get_yaxis().set_visible(False)

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11893491124260355, 'fraction_numerical': 0.016568047337278107, 'mean_word_length': 6.974056603773585, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '226331', 'license': 'MIT', 'n_tokens_mistral': 1299, 'n_tokens_neox': 1228, 'n_words': 295, 'path': 'notebooks/3-autoencoders/ae-exercise-001.ipynb', 'repo_name': 'data-science-workshops/deep-learning', 'size': '3380'}