<jupyter_start><jupyter_text>Generative Adversarial Networks (GANs)  Enable GPUThis notebook and pretty much every other notebook in this repository will run faster if you are using a GPU. On Colab:* Navigate to Edit→Notebook Settings* Select GPU from the Hardware Accelerator drop-downOn Cloud AI Platform Notebooks:* Navigate to https://console.cloud.google.com/ai-platform/notebooks* Create an instance with a GPU or select your instance and add a GPUNext, we'll confirm that we can connect to the GPU with tensorflow:<jupyter_code>import tensorflow as tf
print(tf.version.VERSION)
device_name = tf.test.gpu_device_name()
if device_name != '/device:GPU:0':
    raise SystemError('GPU device not found')
print('Found GPU at: {}'.format(device_name))
# Import libraries and modules
import matplotlib.pyplot as plt
import numpy as np
print(np.__version__)
np.set_printoptions(threshold=np.inf)<jupyter_output>1.18.5<jupyter_text>Vanilla GANs For a vanilla GAN, we use just basic `Dense` layers for both the generator and discriminator.<jupyter_code># Create the generator.
latent_dim = 512
vanilla_generator = tf.keras.Sequential(
    [
        tf.keras.Input(shape=(latent_dim,)),
        tf.keras.layers.Dense(units=256),
        tf.keras.layers.LeakyReLU(alpha=0.2),
        tf.keras.layers.Dense(units=512),
        tf.keras.layers.LeakyReLU(alpha=0.2),
        tf.keras.layers.Dense(units=1024),
        tf.keras.layers.LeakyReLU(alpha=0.2),
        tf.keras.layers.Dense(units=28 * 28 * 1, activation="tanh"),
        tf.keras.layers.Reshape(target_shape=(28, 28, 1))
    ],
    name="vanilla_generator"
)
vanilla_generator.summary()
# Plot generator model.
tf.keras.utils.plot_model(
    model=vanilla_generator,
    to_file="vanilla_generator_model.png",
    show_shapes=True,
    show_layer_names=True
)
# Create the discriminator.
vanilla_discriminator = tf.keras.Sequential(
    [
        tf.keras.Input(shape=(28, 28, 1)),
        tf.keras.layers.Flatten(),
        tf.keras.layers.Dense(units=1024),
        tf.keras.layers.LeakyReLU(alpha=0.2),
        tf.keras.layers.Dense(units=512),
        tf.keras.layers.LeakyReLU(alpha=0.2),
        tf.keras.layers.Dense(units=256),
        tf.keras.layers.LeakyReLU(alpha=0.2),
        tf.keras.layers.Dense(units=1),
    ],
    name="vanilla_discriminator"
)
vanilla_discriminator.summary()
# Plot discriminator model.
tf.keras.utils.plot_model(
    model=vanilla_discriminator,
    to_file="vanilla_discriminator_model.png",
    show_shapes=True,
    show_layer_names=True
)<jupyter_output><empty_output><jupyter_text>We'll create a custom `Model` class named `GAN` that we can use to facilitate training of our generator and discriminator networks.<jupyter_code>class GAN(tf.keras.Model):
    """Custom model for training a GAN.

    Attributes:
        discriminator: Keras `Model`, the discriminator network.
        generator: Keras `Model`, the generator network.
        latent_dim: int, the latent vector dimension length.
        one_sided_label_smoothing: float, how much to smooth the real image
            labels by value in [0., 1.].
        discriminator_optimizer: Keras `Optimizer`, optimizer for
            discriminator network.
        generator_optimizer: Keras `Optimizer`, optimizer for generator
            network.
        loss_fn: Keras `Loss`, loss function for training both networks.
    """
    def __init__(
        self, discriminator, generator, latent_dim, one_sided_label_smoothing
    ):
        """Instantiates `GAN` model class.

        Args:
            discriminator: Keras `Model`, the discriminator network.
            generator: Keras `Model`, the generator network.
            latent_dim: int, the latent vector dimension length.
            one_sided_label_smoothing: float, how much to smooth the real image
                labels by value in [0., 1.].
        """
        super(GAN, self).__init__()
        self.discriminator = discriminator
        self.generator = generator
        self.latent_dim = latent_dim
        self.one_sided_label_smoothing = one_sided_label_smoothing

    def compile(self, discriminator_optimizer, generator_optimizer, loss_fn):
        """Overrides base `Model` class' compile method.

        Args:
            discriminator_optimizer: Keras `Optimizer`, optimizer for
                discriminator network.
            generator_optimizer: Keras `Optimizer`, optimizer for generator
                network.
            loss_fn: Keras `Loss`, loss function for training both networks.
        """
        super(GAN, self).compile()
        self.discriminator_optimizer = discriminator_optimizer
        self.generator_optimizer = generator_optimizer
        self.loss_fn = loss_fn

    def discriminator_train_step(self, batch_size, real_images):
        """Trains discriminator for one step.

        Args:
            batch_size: tensor, scalar tensor representing number of elements
                in the mini-batch.
            real_images: tensor, rank 4 tensor of real images with shape
                (batch_size, height, width, depth).

        Returns:
            discriminator_loss: tensor, scalar loss of disriminator step.
        """
        # Sample random points in the latent space.
        random_latent_vectors = tf.random.normal(
            shape=(batch_size, self.latent_dim)
        )

        # Generate images from noise.
        generated_images = self.generator(inputs=random_latent_vectors)

        # Combine generated images with real images.
        combined_images = tf.concat(
            values=[generated_images, real_images], axis=0
        )

        # Create fake and real labels.
        fake_labels = tf.zeros(shape=(batch_size, 1))
        real_labels = tf.ones(shape=(batch_size, 1))

        # Smooth real labels to help with training.
        real_labels *= self.one_sided_label_smoothing

        # Combine labels to be in-line with combined images.
        labels = tf.concat(
            values=[fake_labels, real_labels], axis=0
        )

        # Train ONLY the discriminator.
        with tf.GradientTape() as tape:
            predictions = self.discriminator(inputs=combined_images)
            discriminator_loss = self.loss_fn(
                y_true=labels, y_pred=predictions
            )

        grads = tape.gradient(
            target=discriminator_loss,
            sources=self.discriminator.trainable_weights
        )

        self.discriminator_optimizer.apply_gradients(
            grads_and_vars=zip(grads, self.discriminator.trainable_weights)
        )

        return discriminator_loss

    def generator_train_step(self, batch_size):
        """Trains generator for one step.

        Args:
            batch_size: tensor, scalar tensor representing number of elements
                in the mini-batch.

        Returns:
            generator_loss: tensor, scalar loss of generator step.
        """
        # Sample random points in the latent space.
        random_latent_vectors = tf.random.normal(
            shape=(batch_size, self.latent_dim)
        )

        # Create labels as if they're real images.
        labels = tf.ones(shape=(batch_size, 1))

        # Train ONLY the generator.
        with tf.GradientTape() as tape:
            predictions = self.discriminator(
                inputs=self.generator(inputs=random_latent_vectors)
            )
            generator_loss = self.loss_fn(y_true=labels, y_pred=predictions)

        grads = tape.gradient(
            target=generator_loss, sources=self.generator.trainable_weights
        )

        self.generator_optimizer.apply_gradients(
            grads_and_vars=zip(grads, self.generator.trainable_weights)
        )

        return generator_loss

    def train_step(self, real_images):
        """Trains `GAN` for one step.

        Args:
            real_images: tensor, rank 4 tensor of real images with shape
                (batch_size, height, width, depth).

        Returns:
            losses: dict, dictionary of losses from each sub-training step.
        """
        if isinstance(real_images, tuple):
            real_images = real_images[0]

        # Get dynamic batch size to use for random noise vectors.
        batch_size = tf.shape(input=real_images)[0]

        # First train discriminator.
        discriminator_loss = self.discriminator_train_step(
            batch_size, real_images
        )

        # Then train generator.
        generator_loss = self.generator_train_step(batch_size)

        losses = {
            "discriminator_loss": discriminator_loss,
            "generator_loss": generator_loss
        }

        return losses

def create_train_dataset(batch_size):
    """Creates training dataset.

    Args:
        batch_size: int, number of elements in a mini-batch.

    Returns:
        dataset: `Dataset`, dataset object for training using MNIST.
    """
    # Get and format MNIST data.
    (x_train, _), (x_test, _) = tf.keras.datasets.mnist.load_data()

    # Let's combine both training and test sets.
    combined = np.concatenate([x_train, x_test])
    combined = (combined.astype("float32") - 127.5) / 255.0
    combined = np.reshape(combined, newshape=(-1, 28, 28, 1))

    # Create tf.data.Dataset for training.
    dataset = tf.data.Dataset.from_tensor_slices(tensors=combined)
    dataset = dataset.shuffle(buffer_size=70000).batch(batch_size=batch_size)

    return dataset

# Instantiate a GAN instance using our vanilla generator and discriminator.
vanilla_gan = GAN(
    discriminator=vanilla_discriminator,
    generator=vanilla_generator,
    latent_dim=latent_dim,
    one_sided_label_smoothing=0.9
)

vanilla_gan.compile(
    discriminator_optimizer=tf.keras.optimizers.Adam(
        learning_rate=0.0002, beta_1=0.5, beta_2=0.999, epsilon=1e-8
    ),
    generator_optimizer=tf.keras.optimizers.Adam(
        learning_rate=0.0002, beta_1=0.5, beta_2=0.999, epsilon=1e-8
    ),
    loss_fn=tf.keras.losses.BinaryCrossentropy(from_logits=True)
)

# Train vanilla GAN model.
vanilla_gan_history = vanilla_gan.fit(
    create_train_dataset(batch_size=128), epochs=30
)<jupyter_output>Epoch 1/30
547/547 [==============================] - 2s 4ms/step - discriminator_loss: 0.6181 - generator_loss: 1.4087
Epoch 2/30
547/547 [==============================] - 2s 4ms/step - discriminator_loss: 0.6057 - generator_loss: 1.3680
Epoch 3/30
547/547 [==============================] - 2s 4ms/step - discriminator_loss: 0.5793 - generator_loss: 1.4029
Epoch 4/30
547/547 [==============================] - 2s 4ms/step - discriminator_loss: 0.5959 - generator_loss: 1.2946
Epoch 5/30
547/547 [==============================] - 2s 4ms/step - discriminator_loss: 0.5994 - generator_loss: 1.2523
Epoch 6/30
547/547 [==============================] - 2s 4ms/step - discriminator_loss: 0.5967 - generator_loss: 1.2620
Epoch 7/30
547/547 [==============================] - 2s 4ms/step - discriminator_loss: 0.5946 - generator_loss: 1.2734
Epoch 8/30
547/547 [==============================] - 2s 4ms/step - discriminator_loss: 0.5902 - generator_loss: 1.2940
Epoch 9/30
547/547 [====================[...]<jupyter_text>Let's plot the loss history and some generated images using our trained model.<jupyter_code>def plot_loss_history(history):
    """Plots loss history.

    Args:
        history: `keras.callbacks.History`, history object from training job.
    """
    plt.plot(history.history["generator_loss"])
    plt.plot(history.history["discriminator_loss"])
    plt.title("Training losses")
    plt.ylabel("Loss")
    plt.xlabel("Epoch")
    plt.legend(["generator", "discriminator"], loc="upper left")
    plt.show()

def plot_images(images):
    """Plots images.

    Args:
        images: np.array, array of images of
            [num_images, image_size, image_size, num_channels].
    """
    num_images = len(images)

    plt.figure(figsize=(20, 20))
    for i in range(num_images):
        image = images[i]
        plt.subplot(1, num_images, i + 1)
        plt.xticks([])
        plt.yticks([])
        plt.grid(False)
        plt.imshow(
            tf.reshape(image, image.shape[:-1]),
            cmap="gray_r"
        )
    plt.show()
plot_loss_history(history=vanilla_gan_history)
plot_images(
    images=vanilla_gan.generator(
        inputs=tf.random.normal(shape=(12, latent_dim))
    )
)<jupyter_output><empty_output><jupyter_text>DCGAN A Deep-convolutional GAN (DCGAN) uses convolutional (`Conv2D`) and deconvolutional (`Conv2DTranspose`) instead of `Dense` layers for the discriminator and generator respectively.<jupyter_code># Create the generator
def create_generator_block(filters, kernel_size, strides, padding, alpha):
    """Creates generator block of layers.

    Args:
        filters: int, number of output filters/channels.
        kernel_size: int/2-tuple of ints, the height and width of the kernel.
        strides: int/2-tuple of ints, the height and width of each stride of
            sliding window.
        padding: str, the type of padding of the output: same or valid.
        alpha: float, the amount of leakiness of LeakyRelu activations.

    Returns:
        List of layers.
    """
    return [
        tf.keras.layers.Conv2DTranspose(
            filters=filters,
            kernel_size=kernel_size,
            strides=strides,
            padding=padding
        ),
        tf.keras.layers.BatchNormalization(),
        tf.keras.layers.LeakyReLU(alpha=alpha)
    ]

def create_dcgan_generator(latent_dim):
    """Creates deep convolutional GAN generator network.

    Args:
        latent_dim: int, the latent vector dimension length.

    Returns:
        Keras Sequential Model.
    """
    dcgan_generator = [
        tf.keras.Input(shape=(latent_dim,)),
        tf.keras.layers.Dense(units=7 * 7 * 256),
        tf.keras.layers.LeakyReLU(alpha=0.2),
        tf.keras.layers.Reshape(target_shape=(7, 7, 256)),
    ] + create_generator_block(
        filters=128, kernel_size=4, strides=2, padding="same", alpha=0.2
    ) + create_generator_block(
        filters=128, kernel_size=4, strides=2, padding="same", alpha=0.2
    ) + [
        tf.keras.layers.Conv2DTranspose(
            filters=1,
            kernel_size=3,
            strides=1,
            padding="same",
            activation="tanh"
        )
    ]

    return tf.keras.Sequential(layers=dcgan_generator, name="dcgan_generator")

dcgan_generator = create_dcgan_generator(latent_dim=512)
dcgan_generator.summary()
# Plot generator model.
tf.keras.utils.plot_model(
    model=dcgan_generator,
    to_file="dcgan_generator_model.png",
    show_shapes=True,
    show_layer_names=True
)
# Create the discriminator
def create_discriminator_block(filters, kernel_size, strides, padding, alpha):
    """Creates discriminator block of layers.

    Args:
        filters: int, number of output filters/channels.
        kernel_size: int/2-tuple of ints, the height and width of the kernel.
        strides: int/2-tuple of ints, the height and width of each stride of
            sliding window.
        padding: str, the type of padding of the output: same or valid.
        alpha: float, the amount of leakiness of LeakyRelu activations.

    Returns:
        List of layers.
    """
    return [
        tf.keras.layers.Conv2D(
            filters=filters,
            kernel_size=kernel_size,
            strides=strides,
            padding=padding
        ),
        tf.keras.layers.BatchNormalization(),
        tf.keras.layers.LeakyReLU(alpha=alpha)
    ]

def create_dcgan_discriminator(input_shape):
    """Creates deep convolutional GAN discriminator network.

    Args:
        input_shape: tuple, the shape of the input without batch dimension.

    Returns:
        Keras Sequential Model.
    """
    dcgan_discriminator = [
        tf.keras.Input(shape=input_shape),
        tf.keras.layers.Conv2D(
            filters=64, kernel_size=3, strides=1, padding="same"
        ),
        tf.keras.layers.LeakyReLU(alpha=0.2)
    ] + create_discriminator_block(
        filters=128, kernel_size=3, strides=2, padding="same", alpha=0.2
    ) + create_discriminator_block(
        filters=128, kernel_size=3, strides=2, padding="same", alpha=0.2
    ) + create_discriminator_block(
        filters=256, kernel_size=3, strides=2, padding="same", alpha=0.2
    ) + [
        tf.keras.layers.Flatten(),
        tf.keras.layers.Dense(units=1)
    ]

    return tf.keras.Sequential(layers=dcgan_discriminator, name="dcgan_discriminator")

dcgan_discriminator = create_dcgan_discriminator(input_shape=(28, 28, 1))
dcgan_discriminator.summary()
# Plot generator model.
tf.keras.utils.plot_model(
    model=dcgan_discriminator,
    to_file="dcgan_discriminator_model.png",
    show_shapes=True,
    show_layer_names=True
)
# Instantiate a GAN instance using our DCGAN generator and discriminator.
dcgan = GAN(
    discriminator=dcgan_discriminator,
    generator=dcgan_generator,
    latent_dim=latent_dim,
    one_sided_label_smoothing=0.9
)

dcgan.compile(
    discriminator_optimizer=tf.keras.optimizers.Adam(
        learning_rate=0.0002, beta_1=0.5, beta_2=0.999, epsilon=1e-8
    ),
    generator_optimizer=tf.keras.optimizers.Adam(
        learning_rate=0.0002, beta_1=0.5, beta_2=0.999, epsilon=1e-8
    ),
    loss_fn=tf.keras.losses.BinaryCrossentropy(from_logits=True)
)

# Train DCGAN model.
dcgan_history = dcgan.fit(create_train_dataset(batch_size=128), epochs=50)<jupyter_output>Epoch 1/50
547/547 [==============================] - 52s 95ms/step - discriminator_loss: 0.5344 - generator_loss: 1.3797
Epoch 2/50
547/547 [==============================] - 51s 94ms/step - discriminator_loss: 0.4646 - generator_loss: 1.7334
Epoch 3/50
547/547 [==============================] - 52s 94ms/step - discriminator_loss: 0.4697 - generator_loss: 1.7260
Epoch 4/50
547/547 [==============================] - 51s 94ms/step - discriminator_loss: 0.4705 - generator_loss: 1.7459
Epoch 5/50
547/547 [==============================] - 52s 94ms/step - discriminator_loss: 0.4673 - generator_loss: 1.7684
Epoch 6/50
547/547 [==============================] - 52s 95ms/step - discriminator_loss: 0.4675 - generator_loss: 1.7803
Epoch 7/50
547/547 [==============================] - 52s 94ms/step - discriminator_loss: 0.4517 - generator_loss: 1.8999
Epoch 8/50
547/547 [==============================] - 52s 94ms/step - discriminator_loss: 0.4392 - generator_loss: 2.0122
Epoch 9/50
547/547 [====[...]<jupyter_text>Let's plot the loss history and some generated images using our trained model.<jupyter_code>plot_loss_history(history=dcgan_history)
plot_images(
    images=dcgan.generator(inputs=tf.random.normal(shape=(12, latent_dim)))
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12347853939782191, 'fraction_numerical': 0.03309844117019005, 'mean_word_length': 3.0216831258050667, 'pattern_counts': {'":': 2, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '370660', 'license': 'Apache-2.0', 'n_tokens_mistral': 6277, 'n_tokens_neox': 5744, 'n_words': 1311, 'path': '12_generation/12c_gan.ipynb', 'repo_name': 'bdwivedi/practical-ml-vision-book', 'size': '18732'}