<jupyter_start><jupyter_text>Denoising Autoencoders And Where To Find ThemIn this programming assignment we're going to train deep autoencoders and apply them to faces and similar images search.Our new test subjects are human faces from the [lfw dataset](http://vis-www.cs.umass.edu/lfw/).<jupyter_code>%%bash

shred -u setup_colab.py

wget https://raw.githubusercontent.com/hse-aml/intro-to-dl-pytorch/main/utils/setup_colab.py -O setup_colab.py
import setup_colab

setup_colab.setup_week04()
from collections import defaultdict

import numpy as np
from matplotlib import pyplot as plt

import download_utils
from lfw_dataset import load_lfw_dataset
from sklearn.model_selection import train_test_split

import torch
from torch import nn
from torch import optim
from torch.utils.data import TensorDataset, DataLoader
class AverageMeter:
    
    def __init__(self):
        self.reset()

    def reset(self):
        self.val = 0
        self.avg = 0
        self.sum = 0
        self.count = 0

    def update(self, val, n=1):
        self.val = val
        self.sum += val * n
        self.count += n
        self.avg = self.sum / self.count<jupyter_output><empty_output><jupyter_text>Fill in your Coursera token and emailTo successfully submit your answers to our grader, please fill in your Coursera submission token and email.<jupyter_code>import grading

grader = grading.Grader(
    assignment_key="<KEY>",
    all_parts=["FtBSK", "83Glu", "fnM1K", "T5tJ7", "UF05M"]
)
# token expires every 30 min
COURSERA_TOKEN = "### YOUR TOKEN HERE ###"
COURSERA_EMAIL = "### YOUR EMAIL HERE ###"<jupyter_output><empty_output><jupyter_text>Load datasetDataset was downloaded for you. Relevant links (just in case):- http://www.cs.columbia.edu/CAVE/databases/pubfig/download/lfw_attributes.txt- http://vis-www.cs.umass.edu/lfw/lfw-deepfunneled.tgz- http://vis-www.cs.umass.edu/lfw/lfw.tgz<jupyter_code># we downloaded them for you, just link them here
download_utils.download_week_4_resources('.')
# load images
X, attr = load_lfw_dataset(use_raw=True, dimx=32, dimy=32)
IMG_SHAPE = X.shape[1:]

# center images
X = X.astype('float32') / 255.0 - 0.5
X = np.transpose(X, (0, 3, 1, 2))

# split
X_train, X_test = train_test_split(X, test_size=0.1, random_state=42)
X_train = torch.from_numpy(X_train)
X_test = torch.from_numpy(X_test)
def show_image(x):
    x = x.permute(1, 2, 0)
    plt.imshow(torch.clamp(x + 0.5, 0, 1))
show_image(X_train[3210])<jupyter_output><empty_output><jupyter_text>Dataloaders<jupyter_code>from torch.utils.data import TensorDataset, DataLoader
BATCH_SIZE = 32

train_dataset = TensorDataset(X_train)
test_dataset = TensorDataset(X_test)


# try changing arguments of DataLoaders to speed up processing
train_dataloader = DataLoader(train_dataset, BATCH_SIZE, num_workers=4)
test_dataloader = DataLoader(test_dataset, BATCH_SIZE, num_workers=4)<jupyter_output><empty_output><jupyter_text>Autoencoder architecture  First step: PCAPrincipial Component Analysis is a popular dimensionality reduction method. Under the hood, PCA attempts to decompose object-feature matrix $X$ into two smaller matrices: $W$ and $\hat W$ minimizing _mean squared error_:$$\|(X W) \hat{W} - X\|^2_2 \to_{W, \hat{W}} \min$$- $X \in \mathbb{R}^{n \times m}$ - object matrix (**centered**);- $W \in \mathbb{R}^{m \times d}$ - matrix of direct transformation;- $\hat{W} \in \mathbb{R}^{d \times m}$ - matrix of reverse transformation;- $n$ samples, $m$ original dimensions and $d$ target dimensions;In geometric terms, we want to find d axes along which most of variance occurs. The "natural" axes, if you wish.PCA can also be seen as a special case of an autoencoder.* __Encoder__: X -> Dense(d units) -> code* __Decoder__: code -> Dense(m units) -> XWhere Dense is a fully-connected layer with linear activaton:   $f(X) = W \cdot X + \vec b $Note: the bias term in those layers is responsible for "centering" the matrix i.e. substracting mean.<jupyter_code>from typing import Tuple


class AutoEncoder(nn.Module):
    
    # note that typing is not necessary
    # but it makes life much easier
    def __init__(self, image_shape: Tuple[int, ...], code_size: int):
        super().__init__()
        
        assert len(image_shape) == 3
        
        self.image_shape = image_shape
        self.code_size = code_size
        
        input_dim = image_shape[0] * image_shape[1] * image_shape[2]
        
        # Initialize `self.encoder` and `self.decoder` as simple Linear layers
        ### YOUR CODE HERE ###
        
    def forward(self, image: torch.Tensor):
        code = self.encode(image)
        return self.decode(code)
    
    def encode(self, image: torch.Tensor):
        image = image.flatten(start_dim=1)
        return self.encoder(image)
    
    def decode(self, code: torch.Tensor):
        image = self.decoder(code)
        image = image.view(-1, *self.image_shape)
        return image<jupyter_output><empty_output><jupyter_text>Init train pipeline<jupyter_code>NUM_EPOCH = 15
DEVICE = torch.device('cuda')

autoencoder = AutoEncoder((3, 32, 32), 100).to(DEVICE)
criterion = nn.MSELoss()
optimizer = optim.Adam(autoencoder.parameters(), lr=1e-3)
history = defaultdict(list)

# TRAIN MODEL
### YOUR CODE HERE ### 
# Feel free to use train loop from notebook of previous week
assert min(history['loss']) < 0.0036, 'Loss is too high!'<jupyter_output><empty_output><jupyter_text>Evaluate model on test dataCalculate loss and vizualize some images<jupyter_code>def visualize(image, autoencoder):
    """
    Draws original, encoded and decoded images
    
    
    NOTE
    Expect image.shape == [3, 32, 32]
    """
    
    with torch.no_grad():
        code = autoencoder.encode(image.unsqueeze(0))
        reco = autoencoder.decode(code).squeeze(0)

    plt.subplot(1, 3, 1)
    plt.title("Original")
    show_image(image.squeeze(0))

    plt.subplot(1, 3, 2)
    plt.title("Code")
    plt.imshow(code.view(code.shape[-1] // 2, -1))

    plt.subplot(1, 3, 3)
    plt.title("Reconstructed")
    show_image(reco)
    plt.show()

autoencoder = autoencoder.cpu()
for i in range(5):
    image, = test_dataset[i]
    visualize(image, autoencoder)
## GRADED PART, DO NOT CHANGE!
grader.set_answer("T5tJ7", min(history['loss']))
# you can make submission with answers so far to check yourself at this stage
grader.submit(COURSERA_EMAIL, COURSERA_TOKEN)<jupyter_output><empty_output><jupyter_text>Going deeper: convolutional autoencoderPCA is neat but surely we can do better. This time we want you to build a deep convolutional autoencoder by... stacking more layers. EncoderThe **encoder** part is pretty standard, we stack convolutional and pooling layers and finish with a dense layer to get the representation of desirable size (`code_size`).We recommend to use `nn.ELU'` for all convolutional and dense layers.We recommend to repeat (conv, pool) 4 times with kernel size (3, 3) (don't forget to add padding=1) and the following numbers of output channels: `32, 64, 128, 256`.Remember to flatten (`nn.Flatten()`) output before adding the last dense layer! DecoderFor **decoder** we will use so-called "transpose convolution". Traditional convolutional layer takes a patch of an image and produces a number (patch -> number). In "transpose convolution" we want to take a number and produce a patch of an image (number -> patch). We need this layer to "undo" convolutions in encoder. We had a glimpse of it during week 3 (watch [this video](https://www.coursera.org/learn/intro-to-deep-learning-mds/lecture/auRqf/a-glimpse-of-other-computer-vision-tasks) starting at 5:41).Here's how "transpose convolution" works:In this example we use a stride of 2 to produce 4x4 output, this way we "undo" pooling as well. Another way to think about it: we "undo" convolution with stride 2 (which is similar to conv + pool).You can add "transpose convolution" layer in PyTorch like this:```pythonnn.ConvTranspose2d(...)```Our decoder starts with a dense layer to "undo" the last layer of encoder. Remember to reshape its output to "undo" `nn.Flatten()` in encoder.Now we're ready to undo (conv, pool) pairs. For this we need to stack 4 `nn.ConvTranspose2d` layers with the following numbers of output channels: `128, 64, 32, 3`. Each of these layers will learn to "undo" (conv, pool) pair in encoder. For the last `nn.ConvTranspose2d` layer don't use any activation because that is our final image.<jupyter_code># Let's play around with transpose convolution on examples first
def test_conv2d_transpose(img_size, filter_size, padding=0):
    print("Transpose convolution test for img_size={}, filter_size={}:".format(img_size, filter_size))
    
    x = torch.arange(img_size ** 2).view(1, 1, img_size, img_size).float()
    
    conv_transpose = nn.ConvTranspose2d(
        in_channels=1, out_channels=1,
        kernel_size=(filter_size, filter_size),
        stride=2, padding=padding, bias=False
    )
    nn.init.ones_(conv_transpose.weight)

    with torch.no_grad():
        result = conv_transpose(x)

    print("input:")
    print(x[0, 0, :, :])
    print("filter:")
    print(conv_transpose.weight[:, :, 0, 0])
    print("output:")
    print(result[0, 0, :, :])
        
test_conv2d_transpose(img_size=2, filter_size=2)
test_conv2d_transpose(img_size=2, filter_size=3)
test_conv2d_transpose(img_size=2, filter_size=3, padding=1)
test_conv2d_transpose(img_size=4, filter_size=2)
test_conv2d_transpose(img_size=4, filter_size=3)
test_conv2d_transpose(img_size=4, filter_size=3, padding=1)
encode_block = lambda i, o: nn.Sequential(
    nn.Conv2d(i, o, kernel_size=3, padding=1),
    nn.ELU(),
    nn.MaxPool2d(2)
)

decoder_block = lambda i, o, k, s, p, a: nn.Sequential(
    nn.ConvTranspose2d(i, o, k, s, p),
    nn.ELU() if a else nn.Identity()
)

class DeepAutoencoder(nn.Module):
    
    def __init__(self, image_shape, code_size):
        super().__init__()
        
        assert image_shape == (3, 32, 32)
        
        # Initialize `self.encoder` with `encode_block` blocks
        ### YOUR CODE HERE ###

        self.projection = nn.Sequential(
            nn.Flatten(start_dim=1),
            nn.Linear(256 * 2 * 2, code_size)
        )
        self.reprojection = nn.Linear(code_size, 256 * 2 * 2)

        # Initialize `self.decoder` with `decoder_block` blocks
        ### YOUR CODE HERE ###

    
    def forward(self, image):
        code = self.encode(image)
        recon = self.decode(code)
        return recon
    
    def encode(self, image):
        output = self.encoder(image)
        code = self.projection(output)
        return code
    
    def decode(self, code):
        output = self.reprojection(code)
        output = output.view(-1, 256, 2, 2)
        image = self.decoder(output)
        return image
IMG_SHAPE = (3, 32, 32)

for code_size in [1, 8, 32, 128, 512]:
    test_input = torch.randn(1, *IMG_SHAPE)
    model = DeepAutoencoder(IMG_SHAPE, code_size)
    print(f"Testing code size {code_size}")
    
    code = model.encode(test_input)
    recon_image = model.decode(code)
    
    assert code.size(-1) == code_size, "encoder must output a code of required size"
    assert recon_image.shape[1:] == IMG_SHAPE
    
    assert len([p for p in model.encoder.parameters()]) >= 6, "encoder must contain at least 3 layers"
    assert len([p for p in model.decoder.parameters()]) >= 6, "decoder must contain at least 3 layers"

print("All tests passed!")
def get_num_params(module):
    return sum(p.numel() for p in module.parameters())
autoencoder = DeepAutoencoder((3, 32, 32), 32)

num_encoder_params = get_num_params(autoencoder.encoder)
num_decoder_params = get_num_params(autoencoder.decoder)
print(num_encoder_params, num_decoder_params)

assert np.abs(num_encoder_params - num_decoder_params) < num_encoder_params * 0.1, \
    "Number of parameters in encoder and decoder should be close"<jupyter_output><empty_output><jupyter_text>Convolutional autoencoder training. This will take **1 hour**. You're aiming at ~0.0056 validation MSE and ~0.0054 training MSE.<jupyter_code>NUM_EPOCH = 20
DEVICE = torch.device('cuda')

deep_autoencoder = DeepAutoencoder((3, 32, 32), 32).to(DEVICE)
criterion = nn.MSELoss()
optimizer = optim.Adam(deep_autoencoder.parameters(), lr=1e-3)
history = defaultdict(list)
### YOUR CODE HERE ###
assert history['loss'][-1] < 0.0055, 'Loss is too high!'<jupyter_output><empty_output><jupyter_text>Evaluate model on test dataCalculate loss and vizualize some images<jupyter_code>deep_autoencoder = deep_autoencoder.cpu()
for i in range(5):
    image, = test_dataset[i]
    visualize(image, deep_autoencoder)
# save trained weights
torch.save(deep_autoencoder.state_dict(), 'deep_autoencoder.pt')
state_dict = torch.load('deep_autoencoder.pt')
deep_autoencoder.load_state_dict(state_dict)<jupyter_output><empty_output><jupyter_text>Submit to Coursera<jupyter_code>test_input = torch.randn(1, *IMG_SHAPE)
model = DeepAutoencoder(IMG_SHAPE, 15)

code = model.encode(test_input)
recon_image = model.decode(code)
## GRADED PART, DO NOT CHANGE!
grader.set_answer("FtBSK", min(history['loss']))
grader.set_answer("83Glu", code.size(-1))
grader.set_answer("fnM1K", recon_image.shape[1:])
# you can make submission with answers so far to check yourself at this stage
grader.submit(COURSERA_EMAIL, COURSERA_TOKEN)<jupyter_output><empty_output><jupyter_text>Denoising AutoencoderIt shows you one useful application of autoencoders: denoising. You can run this code and make sure denoising works :) Let's now turn our model into a denoising autoencoder:We'll keep the model architecture, but change the way it is trained. In particular, we'll corrupt its input data randomly with noise before each epoch.There are many strategies to introduce noise: adding gaussian white noise, occluding with random black rectangles, etc. We will add gaussian white noise.<jupyter_code>def apply_gaussian_noise(X, sigma=0.1):
    """
    adds noise from standard normal distribution with standard deviation sigma
    :param X: image tensor of shape [batch, 3, height, width]
    Returns X + noise.
    """

    ### YOUR CODE HERE ###
# noise tests
theoretical_std = (X_train[:100].std() ** 2 + 0.5 ** 2) ** .5
our_std = apply_gaussian_noise(X_train[:100], sigma=0.5).std()
assert abs(theoretical_std - our_std) < 0.01, \
    "Standard deviation does not match it's required value. Make sure you use sigma as std."
assert abs(apply_gaussian_noise(X_train[:100], sigma=0.5).mean() - X_train[:100].mean()) < 0.01, \
    "Mean has changed. Please add zero-mean noise"
## GRADED PART, DO NOT CHANGE!
grader.set_answer("UF05M", abs(theoretical_std - our_std).item())
# you can make submission with answers so far to check yourself at this stage
grader.submit(COURSERA_EMAIL, COURSERA_TOKEN)
# test different noise scales
plt.subplot(1, 4, 1)
show_image(X_train[0])
plt.subplot(1, 4, 2)
show_image(apply_gaussian_noise(X_train[:1], sigma=0.01)[0])
plt.subplot(1, 4, 3)
show_image(apply_gaussian_noise(X_train[:1], sigma=0.1)[0])
plt.subplot(1, 4, 4)
show_image(apply_gaussian_noise(X_train[:1], sigma=0.5)[0])<jupyter_output><empty_output><jupyter_text>Training will take **1 hour**.<jupyter_code>NUM_EPOCH = 20
DEVICE = torch.device('cuda')

deep_denoiser_autoencoder = DeepAutoencoder((3, 32, 32), 32).to(DEVICE)
criterion = nn.MSELoss()
optimizer = optim.Adam(deep_denoiser_autoencoder.parameters(), lr=1e-3)
history = defaultdict(list)
loss_meter = AverageMeter()

for _ in range(NUM_EPOCH):

    loss_meter.reset()
    for batch in train_dataloader:
        image, = batch
        noised_image = apply_gaussian_noise(image).to(DEVICE)
        image = image.to(DEVICE)

        recon_image = deep_denoiser_autoencoder(noised_image)
        loss = criterion(image, recon_image)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        loss_meter.update(loss.item())
    
    history['loss'].append(loss_meter.avg)
    print(loss_meter.avg)
assert history['loss'][-1] < 0.0057, 'Loss is too high!'<jupyter_output><empty_output><jupyter_text>Evaluate model on test dataCalculate loss and vizualize some images<jupyter_code>deep_denoiser_autoencoder = deep_denoiser_autoencoder.cpu()
for i in range(5):
    image, = test_dataset[i]
    noised_image = apply_gaussian_noise(image)
    visualize(noised_image, deep_denoiser_autoencoder)<jupyter_output><empty_output><jupyter_text>Optional: Image retrieval with autoencodersSo we've just trained a network that converts image into itself imperfectly. This task is not that useful in and of itself, but it has a number of awesome side-effects. Let's see them in action.First thing we can do is image retrieval aka image search. We will give it an image and find similar images in latent space:To speed up retrieval process, one should use Locality Sensitive Hashing on top of encoded vectors. This [technique](https://erikbern.com/2015/07/04/benchmark-of-approximate-nearest-neighbor-libraries.html) can narrow down the potential nearest neighbours of our image in latent space (encoder code). We will caclulate nearest neighbours in brute force way for simplicity.<jupyter_code># restore trained encoder weights

deep_autoencoder = DeepAutoencoder((3, 32, 32), 32)
deep_autoencoder.load_state_dict(torch.load('deep_autoencoder.pt', map_location='cpu'))

for p in deep_autoencoder.parameters():
    p.requires_grad_(False)
images = X_train

# Encode all images
### YOUR CODE HERE ###

assert len(codes) == len(images)
from sklearn.neighbors.unsupervised import NearestNeighbors
nei_clf = NearestNeighbors(metric="euclidean")
nei_clf.fit(codes.detach().numpy())
def get_similar(image, n_neighbors=5):
    assert image.ndim == 3, "image must be [batch, 3, height, width]"

    code = deep_autoencoder.encode(image.unsqueeze(0)).numpy()
    
    (distances, ), (idx, ) = nei_clf.kneighbors(code, n_neighbors=n_neighbors)
    
    return distances, images[idx]
def show_similar(image):
    
    distances, neighbors = get_similar(image, n_neighbors=3)
    
    plt.figure(figsize=[8, 7])
    plt.subplot(1, 4, 1)
    show_image(image)
    plt.title("Original image")
    
    for i in range(3):
        plt.subplot(1, 4, i + 2)
        show_image(neighbors[i])
        plt.title("Dist=%.3f"%distances[i])
    plt.show()<jupyter_output><empty_output><jupyter_text>Cherry-picked examples:<jupyter_code># smiles
show_similar(X_test[247])
# ethnicity
show_similar(X_test[56])
# glasses
show_similar(X_test[63])<jupyter_output><empty_output><jupyter_text>Optional: Cheap image morphing We can take linear combinations of image codes to produce new images with decoder.<jupyter_code># restore trained encoder weights

deep_autoencoder = DeepAutoencoder((3, 32, 32), 32)
deep_autoencoder.load_state_dict(torch.load('deep_autoencoder.pt', map_location='cpu'))

for p in deep_autoencoder.parameters():
    p.requires_grad_(False)
for _ in range(5):
    image1, image2 = X_test[np.random.randint(0, len(X_test), size=2)]

    code1, code2 = deep_autoencoder.encode(torch.stack([image1, image2], dim=0))

    plt.figure(figsize=[14, 8])
    for i, a in enumerate(torch.linspace(0, 1, steps=7)):

        output_code = code1 * (1 - a) + code2 * a
        output_image = deep_autoencoder.decode(output_code.unsqueeze(0)).squeeze(0)

        plt.subplot(1, 7, i + 1)
        show_image(output_image)
        plt.title("a=%.2f"%a)
        
    plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11454789153520982, 'fraction_numerical': 0.02123930302092999, 'mean_word_length': 4.329395604395605, 'pattern_counts': {'":': 0, '<': 76, '<?xml version=': 0, '>': 78, 'https://': 3, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '239605', 'license': 'MIT', 'n_tokens_mistral': 6787, 'n_tokens_neox': 6389, 'n_words': 1785, 'path': '02_PYTHON/week04/week04_autoencoders.ipynb', 'repo_name': 'milaan9/Deep_Learning_Algorithms_from_Scratch', 'size': '19415'}