<jupyter_start><jupyter_text>Generative Adversarial Networks What is Adversarial?![indiana](https://media.giphy.com/media/tivaSkhu8MbKM/giphy.gif)$\min_{\text{boulder}}\max_{\text{indiana}} V(\text{indiana}, \text{boulder}) = \text{distance between them}$Estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model $G$ that captures the data distribution, and a discriminative model $D$ that estimates the probability that a sample came from the training data rather than $G$ (cite Goodfellow2014)![overviewgan](../images/overview.png "https://arxiv.org/pdf/1710.07035.pdf")Image from <NAME>., <NAME>., <NAME>., <NAME>., <NAME>. and <NAME>., 2018. Generative adversarial networks: An overview. IEEE Signal Processing Magazine, 35(1), pp.53-65.  Basic Notations- $x$ will be the **data element**.- $D(x)$ is the discriminator network which outputs the **probability that $x$ is real or generated**. $D(x)$ should be high when $x$ comes from training data and low when $x$ comes from the generator.- $z$ will be a **latent space vector** sampled from a normal distribution. $G(z)$ represents the generator function that which maps the latent vector $z$ to data-space. - The goal of $G$ is to estimate the distribution that the training data comes from ($p_{data}$) so it can generate fake samples from that estimated distribution ($p_g$).- $D(G(z))$ is the probability that the output of the generator $G$ is a real image. $D$ tries to **maximize the probability it correctly classifies reals and fakes** ($\log D(x)$), and $G$ tries to **minimize the probability that $D$ will predict its outputs are fake** $(\log (1 - D(G(x))$)$\underset{G}{\text{min}} \underset{D}{\text{max}}V(D,G) = \mathbb{E}_{x\sim p_{data}(x)}\big[logD(x)\big] + \mathbb{E}_{z\sim p_{z}(z)}\big[log(1-D(G(x)))\big]$![overviewgan](../images/datadist.png "https://arxiv.org/pdf/1710.07035.pdf")During GAN training, the generator is encouraged to produce a distribution of samples, $p_g(x)$ to match that of real data $p_{data}(x)$.In theory, the solution to this minimax game is where $p_g=p_{data}$, and the discriminator guesses randomly if the inputs are real or fake. However, the convergence theory of GANs is still being actively researched and in reality models do not always train to this point.  Main Loop of GANs training![overviewgan](../images/mainloop.png "https://arxiv.org/pdf/1710.07035.pdf")  How to Develop Deep Learning Models with Pytorch?  Type of Layers**Conv Layers**In the convolution layers, the units are organized in feature maps, in which each unit is connected to local patches of the feature maps belonging to the previous layer through a set of weights, called filter bank. All units within a feature map share the same filter bank, different feature maps within the same layer use different filter banks, the arrangement of these has two justifications, on the one hand, data in form of arrays, as in our case, images, local subsets tend to be highly correlated and on the other hand the local statistics of images are invariant to their location.![conv](https://github.com/celiacintas/star_wars_hackathon/raw/8d46effee4e4a82429eb989f017a31c03a6bc2fd/images/Convolution_schematic.gif)Image from http://ufldl.stanford.edu/tutorial/supervised/FeatureExtractionUsingConvolution/<jupyter_code>from torch import nn

nn.Conv2d?

conv = nn.Conv2d(128, 64, 4, 2, 1)<jupyter_output><empty_output><jupyter_text>**MaxPooling**To reduce the dimensionality of feature maps, a pooling layer is located between the convolution layers. The pooling layers eliminate the non-maximum values by calculating an aggregation function, usually using maximum or average on small input regions. The main goal of these layers is to reduce the computational cost in subsequent layers by reducing the size of future feature maps and providing a form of translational invariance.![max](https://github.com/celiacintas/star_wars_hackathon/raw/8d46effee4e4a82429eb989f017a31c03a6bc2fd/images/maxpool.jpg)Image from http://ufldl.stanford.edu/tutorial/supervised/FeatureExtractionUsingConvolution/ - BatchNorm - UpSampling/ConvTranspose2d  Optimization & LossTo take this search for the minimum error to practice we use **Stochastic Gradient Descent**, it consists of showing the input vectors of a subset of training data, compute the outputs, their errors, calculate the gradient for those examples and adjust the weights accordingly. This process is repeated over several subsets of examples until the objective function average stops decreasing.**explain ADAM**![alt text](https://github.com/celiacintas/star_wars_hackathon/raw/8d46effee4e4a82429eb989f017a31c03a6bc2fd/images/saddle_point_evaluation_optimizers.gif) Check out  http://sebastianruder.com/optimizing-gradient-descent/<jupyter_code>from torch import optim
[x for x in dir(optim) if '__' not in x]
G_optimizer = optim.Adam(conv.parameters(),
                         lr=0.0002, betas=(0.5, 0.999))<jupyter_output><empty_output><jupyter_text>Different types of Loss implemented in PyTorch**Binary Cross Entropy**criterion that measures the Binary Cross Entropy between the target and the output, the loss can be defined as:$\ell(x, y)=L=\left\{l_{1}, \ldots, l_{N}\right\}^{\top}, \quad l_{n}=-w_{n}\left[y_{n} \cdot \log x_{n}+\left(1-y_{n}\right) \cdot \log \left(1-x_{n}\right)\right]$Where ...<jupyter_code>[x for x in dir(nn) if x.endswith('Loss')]
loss = nn.BCELoss()<jupyter_output><empty_output><jupyter_text>Data loading and transformations<jupyter_code>from torch.autograd import Variable
from torchvision import datasets
from torchvision import transforms as tfs
from torch.utils import data
ts_list = [tfs.RandomAffine(0., scale=(0.75, 1.), fillcolor=0),
           tfs.Resize((32, 32)),
           tfs.Grayscale(1)]
imagenet_data = datasets.ImageFolder('../data/egyptian_text/',
                                     transform=tfs.Compose(ts_list))

data_loader = data.DataLoader(imagenet_data, batch_size=32, shuffle=True)
import matplotlib.pyplot as plt
import numpy as np
### visualize a few samples

fig = plt.figure()

for i in range(len(imagenet_data)):
    sample = imagenet_data[i + 10]
    ax = plt.subplot(1, 5, i + 1)
    plt.tight_layout()
    plt.imshow(np.asarray(sample[0]))
    # print(np.asarray(sample[0]))
    ax.set_title('Sample {}'.format(i))
    ax.axis('off')

    if i == 4:
        plt.show()
        break<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.110681712784047, 'fraction_numerical': 0.024269593445663935, 'mean_word_length': 5.024208566108007, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '386599', 'license': 'MIT', 'n_tokens_mistral': 2071, 'n_tokens_neox': 1913, 'n_words': 689, 'path': 'notebooks/part-01-why-GANs/01-why-generative.ipynb', 'repo_name': 'NickleDave/generative-neural-nets-tutorial-scipy2020', 'size': '6500'}