<jupyter_start><jupyter_text>Building LSGAN in TensorFlow Implementing LSGAN is same as vanilla GAN expect for the change in the loss function. We will see how to generate images of handwritten digits using LSGAN.  Import librariesFirst, we will import all the necessary libraries:<jupyter_code>import warnings
warnings.filterwarnings('ignore')

import numpy as np
import tensorflow as tf

from tensorflow.examples.tutorials.mnist import input_data
tf.logging.set_verbosity(tf.logging.ERROR)

import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Read the DatasetLoad our MNIST dataset:<jupyter_code>data = input_data.read_data_sets("data/mnist",one_hot=True)<jupyter_output>Extracting data/mnist/train-images-idx3-ubyte.gz
Extracting data/mnist/train-labels-idx1-ubyte.gz
Extracting data/mnist/t10k-images-idx3-ubyte.gz
Extracting data/mnist/t10k-labels-idx1-ubyte.gz<jupyter_text>Defining Generator Generator $G$ takes the noise $z$ as an input and returns an image. We define the generator as simple three layer feed forward network.<jupyter_code>def generator(z,reuse=None):
    
    with tf.variable_scope('generator',reuse=reuse):
        
        hidden1 = tf.layers.dense(inputs=z,units=128,activation=tf.nn.leaky_relu)
        hidden2 = tf.layers.dense(inputs=hidden1,units=128,activation=tf.nn.leaky_relu)
        output = tf.layers.dense(inputs=hidden2,units=784,activation=tf.nn.tanh)
        
        return output<jupyter_output><empty_output><jupyter_text>Defining DiscriminatorWe know that discriminator $D$ returns the probability. i.e it will tell us the probability of the given image being real. We define the discriminator also as a simple three layer feed forward network.<jupyter_code>def discriminator(X,reuse=None):
    
    with tf.variable_scope('discriminator',reuse=reuse):
        
        hidden1 = tf.layers.dense(inputs=X,units=128,activation=tf.nn.leaky_relu)
        hidden2 = tf.layers.dense(inputs=hidden1,units=128,activation=tf.nn.leaky_relu)
        logits = tf.layers.dense(inputs=hidden2,units=1)
        output = tf.sigmoid(logits)
        
        return logits<jupyter_output><empty_output><jupyter_text>Define the input placeholdersNow, we define the placeholder for the input $x$ and the noise $z$:<jupyter_code>x = tf.placeholder(tf.float32,shape=[None,784])
z = tf.placeholder(tf.float32,shape=[None,100])<jupyter_output><empty_output><jupyter_text>Start the GAN! First we feed the noise z to the generator and it will output the fake image. i.e $ fake \; x = G(z) $<jupyter_code>fake_x = generator(z)<jupyter_output><empty_output><jupyter_text>Now, we feed the real image to the discriminator $D(x)$ and get the probabillty of being real:<jupyter_code>D_logits_real = discriminator(x)<jupyter_output><empty_output><jupyter_text>Similarly, we feed the fake image to the discirminator $D(z)$ and get the probabillty of it being real:<jupyter_code>D_logits_fake = discriminator(fake_x,reuse=True)<jupyter_output><empty_output><jupyter_text>Computing Loss FunctionNow, we will see, how to compute the loss function. Discriminator LossDiscriminator loss is given as:$L^D =\frac{1}{2} \mathbb{E}_{\boldsymbol{x} \sim p_{\mathrm{data}}(\boldsymbol{x})}\left[(D(\boldsymbol{x})-1)^{2}\right]+\frac{1}{2} \mathbb{E}_{\boldsymbol{z} \sim p_{\boldsymbol{z}}(\boldsymbol{z})}\left[(D(G(\boldsymbol{z})))^{2}\right] $First we will implement the first term i.e $\frac{1}{2} \mathbb{E}_{\boldsymbol{x} \sim p_{\mathrm{data}}(\boldsymbol{x})}\left[(D(\boldsymbol{x})-1)^{2}\right]$<jupyter_code>D_loss_real = 0.5*tf.reduce_mean(tf.square(D_logits_real-1))<jupyter_output><empty_output><jupyter_text>Now, the second term, $\frac{1}{2} \mathbb{E}_{\boldsymbol{z} \sim p_{\boldsymbol{z}}(\boldsymbol{z})}\left[(D(G(\boldsymbol{z})))^{2}\right] $<jupyter_code>D_loss_fake = 0.5*tf.reduce_mean(tf.square(D_logits_fake))<jupyter_output><empty_output><jupyter_text>The final discriminator loss can be written as:<jupyter_code>D_loss = D_loss_real + D_loss_fake<jupyter_output><empty_output><jupyter_text>Generator LossGenerator loss is given as:$L^G =\frac{1}{2} \mathbb{E}_{\boldsymbol{z} \sim p_{\boldsymbol{z}}(\boldsymbol{z})}\left[(D(G(\boldsymbol{z}))-1)^{2}\right] $<jupyter_code>G_loss = 0.5*tf.reduce_mean(tf.square(D_logits_fake-1))<jupyter_output><empty_output><jupyter_text>Optimizing the LossNow we need to optimize our generator and discriminator. So, we collect the parameters of the discriminator and generator as $\theta_D$ and $\theta_G$ respectively.<jupyter_code>training_vars = tf.trainable_variables()

theta_D = [var for var in training_vars if 'discriminator' in var.name]
theta_G = [var for var in training_vars if 'generator' in var.name]<jupyter_output><empty_output><jupyter_text>Optimize the loss using adam optimizer:<jupyter_code>D_optimizer = tf.train.AdamOptimizer(0.001).minimize(D_loss,var_list = theta_D)
G_optimizer = tf.train.AdamOptimizer(0.001).minimize(G_loss, var_list = theta_G)<jupyter_output><empty_output><jupyter_text>Begin Training Define the batch size, number of epochs and start the tensorflow session.<jupyter_code>batch_size = 100
num_epochs = 10000

session = tf.InteractiveSession()
session.run(tf.global_variables_initializer())<jupyter_output><empty_output><jupyter_text>Generate Handwritten DigitsStart the tensorflow session and generate handwritten digits:<jupyter_code>#for every epcochs
for epoch in range(num_epochs):
    
    #select number of batches
    num_batches = data.train.num_examples // batch_size
    
    #for every batch
    for i in range(num_batches):
        
        #sample batch of data
        batch = data.train.next_batch(batch_size)
        
        #reshape the input images
        batch_images = batch[0].reshape((batch_size,784))
        batch_images = batch_images * 2 - 1
        
        #sample batch of noise
        batch_noise = np.random.uniform(-1,1,size=(batch_size,100))
        
        #define the feed dictionaries with input x as batch_images and noise z as batch noise
        feed_dict = {x: batch_images, z : batch_noise}
    
        #train the discriminator and generator
        _ = session.run(D_optimizer,feed_dict = feed_dict)
        _ = session.run(G_optimizer,feed_dict = feed_dict)

        
    #compute loss of discriminator and generator
    discriminator_loss = D_loss.eval(feed_dict)
    generator_loss = G_loss.eval(feed_dict)
    
    #feed the noise to a generator on every 100th epoch and generate an image
    if epoch%100==0:
        print("Epoch: {}, Discriminator Loss:{}, Generator Loss: {}".format(epoch,discriminator_loss,generator_loss))

        noise = np.random.uniform(-1,1,size=(1,100))
        _fake_x = fake_x.eval(feed_dict ={z:noise})
        
        plt.imshow(_fake_x.reshape(28,28))
        plt.show()<jupyter_output>Epoch: 0, Discriminator Loss:0.0160291660577, Generator Loss: 0.470727235079
{'fraction_non_alphanumeric': 0.11593573599652628, 'fraction_numerical': 0.020263424518743668, 'mean_word_length': 5.208445642407907, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 63, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '598488', 'license': 'MIT', 'n_tokens_mistral': 2421, 'n_tokens_neox': 2306, 'n_words': 501, 'path': 'Chapter08/8.09 Building LSGAN in TensorFlow.ipynb', 'repo_name': 'bharlow058/Packt-Deep-Learning', 'size': '6909'}