<jupyter_start><jupyter_text>GAN for MNISTRecently I read about GANs and I was blown away by the iheir capabilities. The idea of GAN is simple and beautiful. I am following this [tutorial](https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html). Lets learn more about them!!!  Importing modules Importing the important modules we need.<jupyter_code>from __future__ import print_function
import argparse 
import os
import random
import torch 
import torch.nn as nn
import torch.nn.parallel
import torch.backends.cudnn as cudnn
import torch.optim as optim
import torch.utils.data
import torchvision.datasets as dset
import torchvision.transforms as transforms
import torchvision.utils as vutils
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation
from IPython.display import HTML
%matplotlib inline

# Set random seed for reproducibility
manualSeed = 999
#manualSeed = random.randint(1, 10000) # use if you want new results
print("Random Seed: ", manualSeed)
random.seed(manualSeed)
torch.manual_seed(manualSeed)<jupyter_output>Random Seed:  999<jupyter_text>Defining the Inputs to the network<jupyter_code>dataroot = './data/celeba'  #Root Directory for the dataset 
workers = 2  #Number of threads for loading data with DataLoader 
batch_size = 128  #batch size during training 
image_size = 28  #The image is 64x64
nc = 1 #Number of Channels. 

nz = 100 #Size of latent vector imput to the generator. 
ngf = 64 #Relates to the depth of feature maps carried thorugh generator. 
ndf = 64 #Sets the depth of feature maps propagated through the discriminator. 

num_epochs = 300
lr = 0.0002
beta1 = 0.5 #For adam optimizers
ngpu = 1 #num of gpus used<jupyter_output><empty_output><jupyter_text>Loading the Datatorch.utils.data.DataLoader gives us a python iterable for the dataset. Also checkout how we loaded the data in the DCGAN tutorials from a folder.<jupyter_code>def mnist_data():
    compose = transforms.Compose(
        [transforms.ToTensor(),
         transforms.Normalize([0.5], [0.5])
        ])
    out_dir = './data'
    return dset.MNIST(root=out_dir, train=True, transform=compose, download=True)# Load data
dataset = mnist_data()

dataloader = torch.utils.data.DataLoader(dataset, batch_size=batch_size,
                                         shuffle=True, num_workers=workers)


device = torch.device("cuda:0" if (torch.cuda.is_available() and ngpu > 0) else "cpu")

# Plot some training images
real_batch = next(iter(dataloader))
plt.figure(figsize=(8,8))
plt.axis("off")
plt.title("Training Images")
plt.imshow(np.transpose(vutils.make_grid(real_batch[0].to(device)[:64], padding=2, normalize=True).cpu(),(1,2,0)))<jupyter_output><empty_output><jupyter_text>Weight InitialisationThe authors of DCGAN have specified to initialize all the model weights using a normal distribution with mean = 0, standard_deviation = 0.02Look how beautifully the tutorial has implemented this. These are the small things in pytorch that we can learn.<jupyter_code>def weights_init(m):
    classname = m.__class__.__name__  # We get the name of the class from the object!
    if classname.find('Conv') != -1:
        nn.init.normal_(m.weight.data, 0.0, 0.02)
    elif classname.find('BatchNorm') != -1:
        nn.init.normal_(m.weight.data, 1.0, 0.02)
        nn.init.constant_(m.bias.data, 0)<jupyter_output><empty_output><jupyter_text>Generator So our generator generates the fake samples from a [latent space](https://towardsdatascience.com/understanding-latent-space-in-machine-learning-de5a7c687d8d) vector. Basically the latent space is a compressed encoding of complex information. Using this information we want to get actual data. Generator helps us in doing that. The input tensor to the generator is of shape (num_tensors, tensor_shape, 1, 1). We project our tensor as having nz channels with 1x1 pixels in each channel. nn.ReLU(inplace = True) this means perform x = nn.ReLU(x)<jupyter_code>class Generator(nn.Module) :
    def __init__(self, ngpu) :
        super(Generator, self).__init__()
        self.ngpu = ngpu
        self.main = nn.Sequential(
            nn.Linear(nz, 128), 
            nn.LeakyReLU(0.2, inplace=True),
            nn.Linear(128, 256),
            nn.LeakyReLU(0.2, True), 
            nn.Linear(256, 512),
            nn.LeakyReLU(0.2, True),
            nn.Linear(512, image_size*image_size)
        )
    def forward(self, input):
        input = input.view(input.size(0), nz)
        return self.main(input)

netG = Generator(ngpu).to(device)
netG(torch.zeros(2, nz, 1, 1).to(device)).shape  # Sample Back Propagation 

# netG.apply(weights_init)<jupyter_output><empty_output><jupyter_text>Discriminator The discriminator is a binary classification network whose job is to distinguish between real samples and the fake samples( generated by generator).<jupyter_code>class Discriminator(nn.Module):
    def __init__(self, ngpu):
        super(Discriminator, self).__init__()
        self.ngpu = ngpu
        self.main = nn.Sequential(
                    nn.Linear(image_size*image_size, 512), 
                    nn.LeakyReLU(0.2, True), 
                    nn.Linear(512, 256),
                    nn.LeakyReLU(0.2, True),
                    nn.Linear(256, 128), 
                    nn.LeakyReLU(0.2, True), 
                    nn.Linear(128, 1), 
                    nn.Sigmoid()
        )
    
    def forward(self, input):
        input = input.view(input.size(0), image_size*image_size*nc)
        return self.main(input)

netD = Discriminator(ngpu).to(device)
netD.apply(weights_init)
netD(x).shape<jupyter_output><empty_output><jupyter_text>Loss Functions and OptimisersWe use the binary cross entropy loss here. The optimizer used is adam.<jupyter_code>criterion = nn.BCELoss()

fixed_noise = torch.randn(64, nz, 1, 1, device=device)

# Convention for real and fake labels. 
real_label = 1
fake_label = 0

optimizerD = optim.Adam(netD.parameters(), lr=lr, betas=(beta1, 0.999))
optimizerG = optim.Adam(netG.parameters(), lr=lr, betas=(beta1, 0.999))<jupyter_output><empty_output><jupyter_text>Training the GANSeems like you have to be very careful while training a GAN. [Here](https://github.com/soumith/ganhacks) is a very popular resource.<jupyter_code># Lists to keep track of the progresss
img_list = []
G_losses = []
D_losses = []
iters = 0

print("Starting the training loop!!")
for epoch in range(num_epochs) :
    for i, data in enumerate(dataloader, 0):
         ############################
        # (1) Update D network: maximize log(D(x)) + log(1 - D(G(z)))
        ###########################
        netD.zero_grad()

        real_cpu = data[0].to(device)
        b_size = real_cpu.size(0)
        label = torch.full((b_size,), real_label, device=device)
        output = netD(real_cpu).view(-1)
        errD_real = criterion(output, label)
        errD_real.backward()
        D_x = output.mean().item()
        
        
        
        
        
        
        noise = torch.randn(b_size, nz, 1, 1, device=device)
        fake = netG(noise)
        label.fill_(fake_label)
        output = netD(fake.detach()).view(-1)
        errD_fake = criterion(output, label)
        errD_fake.backward()
        D_G_z1 = output.mean().item()
        errD = errD_real + errD_fake
        optimizerD.step()
        
        
        netG.zero_grad()
        label.fill_(real_label)  # fake labels are real for generator cost
        # Since we just updated D, perform another forward pass of all-fake batch through D
        output = netD(fake).view(-1)
        # Calculate G's loss based on this output
        errG = criterion(output, label)
        # Calculate gradients for G
        errG.backward()
        D_G_z2 = output.mean().item()
        # Update G
        optimizerG.step()
        
        
        # Output training stats
        if i % 50 == 0:
            print('[%d/%d][%d/%d]\tLoss_D: %.4f\tLoss_G: %.4f\tD(x): %.4f\tD(G(z)): %.4f / %.4f'
                  % (epoch, num_epochs, i, len(dataloader),
                     errD.item(), errG.item(), D_x, D_G_z1, D_G_z2))
            noise = torch.randn(1, nz, 1, 1, device=device)
            fake = netG(noise)
            fakenp = fake.cpu().detach().numpy()
            fakenp = fakenp.reshape(28,28)
            plt.imshow(fakenp, cmap='gray')

        # Save Losses for plotting later
        G_losses.append(errG.item())
        D_losses.append(errD.item())

        # Check how the generator is doing by saving G's output on fixed_noise
        if (iters % 500 == 0) or ((epoch == num_epochs-1) and (i == len(dataloader)-1)):
            with torch.no_grad():
                fake = netG(fixed_noise).detach().cpu()
            img_list.append(vutils.make_grid(fake, padding=2, normalize=True))

        iters += 1
        
        
noise = torch.randn(1, nz, 1, 1, device=device)
fake = netG(torch.zeros(1, nz, 1, 1, device=device))
fakenp = fake.cpu().detach().numpy()
fakenp = fakenp.reshape(28,28)
plt.imshow(fakenp, cmap='gray')<jupyter_output><empty_output><jupyter_text>Saving the Models<jupyter_code>torch.save(netG.state_dict(), './models/Generator.pth')
torch.save(netD.state_dict(), './models/Discriminator.pth')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11707422635793119, 'fraction_numerical': 0.021856740965159056, 'mean_word_length': 3.115316117542297, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 37, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '58666', 'license': 'MIT', 'n_tokens_mistral': 3208, 'n_tokens_neox': 2989, 'n_words': 791, 'path': 'Deep_Learning/GANs/GAN for MNIST.ipynb', 'repo_name': 'qureshinomaan/Robotics-CV-Algorithms', 'size': '9242'}