<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generative Adversarial Network\n",
    "\n",
    "In this notebook, we'll be building a generative adversarial network (GAN) trained on the MNIST dataset. From this, we'll be able to generate new handwritten digits!\n",
    "\n",
    "GANs were [first reported on](https://arxiv.org/abs/1406.2661) in 2014 from <NAME> and others in Yoshua Bengio's lab. Since then, GANs have exploded in popularity. Here are a few examples to check out:\n",
    "\n",
    "* [Pix2Pix](https://affinelayer.com/pixsrv/) \n",
    "* [CycleGAN](https://github.com/junyanz/CycleGAN)\n",
    "* [A whole list](https://github.com/wiseodd/generative-models)\n",
    "\n",
    "The idea behind GANs is that you have two networks, a generator $G$ and a discriminator $D$, competing against each other. The generator makes fake data to pass to the discriminator. The discriminator also sees real data and predicts if the data it's received is real or fake. The generator is trained to fool the discriminator, it wants to output data that looks _as close as possible_ to real data. And the discriminator is trained to figure out which data is real and which is fake. What ends up happening is that the generator learns to make data that is indistiguishable from real data to the discriminator.\n",
    "\n",
    "![GAN diagram](assets/gan_diagram.png)\n",
    "\n",
    "The general structure of a GAN is shown in the diagram above, using MNIST images as data. The latent sample is a random vector the generator uses to contruct it's fake images. As the generator learns through training, it figures out how to map these random vectors to recognizable images that can foold the discriminator.\n",
    "\n",
    "The output of the discriminator is a sigmoid function, where 0 indicates a fake image and 1 indicates an real image. If you're interested only in generating new images, you can throw out the discriminator after training. Now, let's see how we build this thing in TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import pickle as pkl\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.\n",
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets('MNIST_data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Inputs\n",
    "\n",
    "First we need to create the inputs for our graph. We need two inputs, one for the discriminator and one for the generator. Here we'll call the discriminator input `inputs_real` and the generator input `inputs_z`. We'll assign them the appropriate sizes for each of the networks.\n",
    "\n",
    ">**Exercise:** Finish the `model_inputs` function below. Create the placeholders for `inputs_real` and `inputs_z` using the input sizes `real_dim` and `z_dim` respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def model_inputs(real_dim, z_dim):\n",
    "    inputs_real = tf.placeholder(tf.float32, shape=(None, real_dim))\n",
    "    inputs_z = tf.placeholder(tf.float32, shape=(None, z_dim))\n",
    "    \n",
    "    return inputs_real, inputs_z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generator network\n",
    "\n",
    "![GAN Network](assets/gan_network.png)\n",
    "\n",
    "Here we'll build the generator network. To make this network a universal function approximator, we'll need at least one hidden layer. We should use a leaky ReLU to allow gradients to flow backwards through the layer unimpeded. A leaky ReLU is like a normal ReLU, except that there is a small non-zero output for negative input values.\n",
    "\n",
    "#### Variable Scope\n",
    "Here we need to use `tf.variable_scope` for two reasons. Firstly, we're going to make sure all the variable names start with `generator`. Similarly, we'll prepend `discriminator` to the discriminator variables. This will help out later when we're training the separate networks.\n",
    "\n",
    "We could just use `tf.name_scope` to set the names, but we also want to reuse these networks with different inputs. For the generator, we're going to train it, but also _sample from it_ as we're training and after training. The discriminator will need to share variables between the fake and real input images. So, we can use the `reuse` keyword for `tf.variable_scope` to tell TensorFlow to reuse the variables instead of creating new ones if we build the graph again.\n",
    "\n",
    "To use `tf.variable_scope`, you use a `with` statement:\n",
    "```python\n",
    "with tf.variable_scope('scope_name', reuse=False):\n",
    "    # code here\n",
    "```\n",
    "\n",
    "Here's more from [the TensorFlow documentation](https://www.tensorflow.org/programmers_guide/variable_scope#the_problem) to get another look at using `tf.variable_scope`.\n",
    "\n",
    "#### Leaky ReLU\n",
    "TensorFlow doesn't provide an operation for leaky ReLUs, so we'll need to make one . For this you can use take the outputs from a linear fully connected layer and pass them to `tf.maximum`. Typically, a parameter `alpha` sets the magnitude of the output for negative values. So, the output for negative input (`x`) values is `alpha*x`, and the output for positive `x` is `x`:\n",
    "$$\n",
    "f(x) = max(\\alpha * x, x)\n",
    "$$\n",
    "\n",
    "#### Tanh Output\n",
    "The generator has been found to perform the best with $tanh$ for the generator output. This means that we'll have to rescale the MNIST images to be between -1 and 1, instead of 0 and 1.\n",
    "\n",
    ">**Exercise:** Implement the generator network in the function below. You'll need to return the tanh output. Make sure to wrap your code in a variable scope, with 'generator' as the scope name, and pass the `reuse` keyword argument from the function to `tf.variable_scope`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def generator(z, out_dim, n_units=128, reuse=False,  alpha=0.01):\n",
    "    ''' Build the generator network.\n",
    "    \n",
    "        Arguments\n",
    "        ---------\n",
    "        z : Input tensor for the generator\n",
    "        out_dim : Shape of the generator output\n",
    "        n_units : Number of units in hidden layer\n",
    "        reuse : Reuse the variables with tf.variable_scope\n",
    "        alpha : leak parameter for leaky ReLU\n",
    "        \n",
    "        Returns\n",
    "        -------\n",
    "        out, logits: \n",
    "    '''\n",
    "    with tf.variable_scope('generator', reuse=reuse):\n",
    "        # Hidden layer\n",
    "        zshape = z.get_shape().as_list()\n",
    "        print('zshape', zshape)\n",
    "        gw1 = tf.Variable(tf.random_normal([zshape[1], n_units], stddev=0.1), name=\"weight\")\n",
    "        gb1 = tf.Variable(tf.zeros([n_units]), name=\"bias\") \n",
    "        h1 = tf.matmul(z, gw1) + gb1\n",
    "        # Leaky ReLU\n",
    "        h1 = tf.maximum(alpha*h1, h1)\n",
    "        \n",
    "        # Logits and tanh output\n",
    "        gwo = tf.Variable(tf.random_normal([n_units, out_dim], stddev=0.1), name=\"output_weight\")\n",
    "        gbo = tf.Variable(tf.zeros([out_dim]), name=\"output_bias\")\n",
    "        logits = tf.matmul(h1, gwo) + gbo\n",
    "        out = tf.tanh(logits)\n",
    "        \n",
    "        return out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discriminator\n",
    "\n",
    "The discriminator network is almost exactly the same as the generator network, except that we're using a sigmoid output layer.\n",
    "\n",
    ">**Exercise:** Implement the discriminator network in the function below. Same as above, you'll need to return both the logits and the sigmoid output. Make sure to wrap your code in a variable scope, with 'discriminator' as the scope name, and pass the `reuse` keyword argument from the function arguments to `tf.variable_scope`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def discriminator(x, n_units=128, reuse=False, alpha=0.01):\n",
    "    ''' Build the discriminator network.\n",
    "    \n",
    "        Arguments\n",
    "        ---------\n",
    "        x : Input tensor for the discriminator\n",
    "        n_units: Number of units in hidden layer\n",
    "        reuse : Reuse the variables with tf.variable_scope\n",
    "        alpha : leak parameter for leaky ReLU\n",
    "        \n",
    "        Returns\n",
    "        -------\n",
    "        out, logits: \n",
    "    '''\n",
    "    with tf.variable_scope('discriminator', reuse=reuse):\n",
    "        # Hidden layer\n",
    "        xshape = x.get_shape().as_list()\n",
    "        print('xshape', xshape)\n",
    "        dw1 = tf.Variable(tf.random_normal([xshape[1], n_units], stddev=0.1), name=\"weight\")\n",
    "        db1 = tf.Variable(tf.zeros([n_units]), name=\"bias\") \n",
    "        h1 = tf.matmul(x, dw1) + db1\n",
    "        # Leaky ReLU\n",
    "        h1 = tf.maximum(alpha*h1, h1)\n",
    "        \n",
    "        dwo = tf.Variable(tf.random_normal([n_units, 1], stddev=0.1), name=\"output_weight\")\n",
    "        dbo = tf.Variable(tf.zeros([1]), name=\"output_bias\")\n",
    "        logits = tf.matmul(h1, dwo) + dbo\n",
    "        out = tf.sigmoid(logits)\n",
    "        \n",
    "        return out, logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Size of input image to discriminator\n",
    "input_size = 784 # 28x28 MNIST images flattened\n",
    "# Size of latent vector to generator\n",
    "z_size = 100\n",
    "# Sizes of hidden layers in generator and discriminator\n",
    "g_hidden_size = 128\n",
    "d_hidden_size = 128\n",
    "# Leak factor for leaky ReLU\n",
    "alpha = 0.01\n",
    "# Label smoothing \n",
    "smooth = 0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build network\n",
    "\n",
    "Now we're building the network from the functions defined above.\n",
    "\n",
    "First is to get our inputs, `input_real, input_z` from `model_inputs` using the sizes of the input and z.\n",
    "\n",
    "Then, we'll create the generator, `generator(input_z, input_size)`. This builds the generator with the appropriate input and output sizes.\n",
    "\n",
    "Then the discriminators. We'll build two of them, one for real data and one for fake data. Since we want the weights to be the same for both real and fake data, we need to reuse the variables. For the fake data, we're getting it from the generator as `g_model`. So the real data discriminator is `discriminator(input_real)` while the fake discriminator is `discriminator(g_model, reuse=True)`.\n",
    "\n",
    ">**Exercise:** Build the network from the functions you defined earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "zshape [None, 100]\n",
      "xshape [None, 784]\n",
      "xshape [None, 100]\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "# Create our input placeholders\n",
    "input_real, input_z = model_inputs(input_size, z_size)\n",
    "\n",
    "# Generator network here\n",
    "g_model = generator(input_z, input_size, g_hidden_size, False, alpha)\n",
    "# g_model is the generator output\n",
    "\n",
    "# Disriminator network here\n",
    "d_model_real, d_logits_real = discriminator(input_real, d_hidden_size, False, alpha)\n",
    "d_model_fake, d_logits_fake = discriminator(input_z, d_hidden_size, True, alpha)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discriminator and Generator Losses\n",
    "\n",
    "Now we need to calculate the losses, which is a little tricky. For the discriminator, the total loss is the sum of the losses for real and fake images, `d_loss = d_loss_real + d_loss_fake`. The losses will by sigmoid cross-entropys, which we can get with `tf.nn.sigmoid_cross_entropy_with_logits`. We'll also wrap that in `tf.reduce_mean` to get the mean for all the images in the batch. So the losses will look something like \n",
    "\n",
    "```python\n",
    "tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=logits, labels=labels))\n",
    "```\n",
    "\n",
    "For the real image logits, we'll use `d_logits_real` which we got from the discriminator in the cell above. For the labels, we want them to be all ones, since these are all real images. To help the discriminator generalize better, the labels are reduced a bit from 1.0 to 0.9, for example,  using the parameter `smooth`. This is known as label smoothing, typically used with classifiers to improve performance. In TensorFlow, it looks something like `labels = tf.ones_like(tensor) * (1 - smooth)`\n",
    "\n",
    "The discriminator loss for the fake data is similar. The logits are `d_logits_fake`, which we got from passing the generator output to the discriminator. These fake logits are used with labels of all zeros. Remember that we want the discriminator to output 1 for real images and 0 for fake images, so we need to set up the losses to reflect that.\n",
    "\n",
    "Finally, the generator losses are using `d_logits_fake`, the fake image logits. But, now the labels are all ones. The generator is trying to fool the discriminator, so it wants to discriminator to output ones for fake images.\n",
    "\n",
    ">**Exercise:** Calculate the losses for the discriminator and the generator. There are two discriminator losses, one for real images and one for fake images. For the real image loss, use the real logits and (smoothed) labels of ones. For the fake image loss, use the fake logits with labels of all zeros. The total discriminator loss is the sum of those two losses. Finally, the generator loss again uses the fake logits from the discriminator, but this time the labels are all ones because the generator wants to fool the discriminator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Calculate losses\n",
    "d_loss_real = \n",
    "\n",
    "d_loss_fake = \n",
    "\n",
    "d_loss = \n",
    "\n",
    "g_loss = "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimizers\n",
    "\n",
    "We want to update the generator and discriminator variables separately. So we need to get the variables for each part build optimizers for the two parts. To get all the trainable variables, we use `tf.trainable_variables()`. This creates a list of all the variables we've defined in our graph.\n",
    "\n",
    "For the generator optimizer, we only want to generator variables. Our past selves were nice and used a variable scope to start all of our generator variable names with `generator`. So, we just need to iterate through the list from `tf.trainable_variables()` and keep variables to start with `generator`. Each variable object has an attribute `name` which holds the name of the variable as a string (`var.name == 'weights_0'` for instance). \n",
    "\n",
    "We can do something similar with the discriminator. All the variables in the discriminator start with `discriminator`.\n",
    "\n",
    "Then, in the optimizer we pass the variable lists to `var_list` in the `minimize` method. This tells the optimizer to only update the listed variables. Something like `tf.train.AdamOptimizer().minimize(loss, var_list=var_list)` will only train the variables in `var_list`.\n",
    "\n",
    ">**Exercise: ** Below, implement the optimizers for the generator and discriminator. First you'll need to get a list of trainable variables, then split that list into two lists, one for the generator variables and another for the discriminator variables. Finally, using `AdamOptimizer`, create an optimizer for each network that update the network variables separately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Optimizers\n",
    "learning_rate = 0.002\n",
    "\n",
    "# Get the trainable_variables, split into G and D parts\n",
    "t_vars = \n",
    "g_vars = \n",
    "d_vars = \n",
    "\n",
    "d_train_opt = \n",
    "g_train_opt = "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "batch_size = 100\n",
    "epochs = 100\n",<commit_msg>Training fails during sampling the generator.<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generative Adversarial Network\n",
    "\n",
    "In this notebook, we'll be building a generative adversarial network (GAN) trained on the MNIST dataset. From this, we'll be able to generate new handwritten digits!\n",
    "\n",
    "GANs were [first reported on](https://arxiv.org/abs/1406.2661) in 2014 from <NAME> and others in Yoshua Bengio's lab. Since then, GANs have exploded in popularity. Here are a few examples to check out:\n",
    "\n",
    "* [Pix2Pix](https://affinelayer.com/pixsrv/) \n",
    "* [CycleGAN](https://github.com/junyanz/CycleGAN)\n",
    "* [A whole list](https://github.com/wiseodd/generative-models)\n",
    "\n",
    "The idea behind GANs is that you have two networks, a generator $G$ and a discriminator $D$, competing against each other. The generator makes fake data to pass to the discriminator. The discriminator also sees real data and predicts if the data it's received is real or fake. The generator is trained to fool the discriminator, it wants to output data that looks _as close as possible_ to real data. And the discriminator is trained to figure out which data is real and which is fake. What ends up happening is that the generator learns to make data that is indistiguishable from real data to the discriminator.\n",
    "\n",
    "![GAN diagram](assets/gan_diagram.png)\n",
    "\n",
    "The general structure of a GAN is shown in the diagram above, using MNIST images as data. The latent sample is a random vector the generator uses to contruct it's fake images. As the generator learns through training, it figures out how to map these random vectors to recognizable images that can foold the discriminator.\n",
    "\n",
    "The output of the discriminator is a sigmoid function, where 0 indicates a fake image and 1 indicates an real image. If you're interested only in generating new images, you can throw out the discriminator after training. Now, let's see how we build this thing in TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import pickle as pkl\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets('MNIST_data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Inputs\n",
    "\n",
    "First we need to create the inputs for our graph. We need two inputs, one for the discriminator and one for the generator. Here we'll call the discriminator input `inputs_real` and the generator input `inputs_z`. We'll assign them the appropriate sizes for each of the networks.\n",
    "\n",
    ">**Exercise:** Finish the `model_inputs` function below. Create the placeholders for `inputs_real` and `inputs_z` using the input sizes `real_dim` and `z_dim` respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def model_inputs(real_dim, z_dim):\n",
    "    inputs_real = tf.placeholder(tf.float32, shape=(None, real_dim))\n",
    "    inputs_z = tf.placeholder(tf.float32, shape=(None, z_dim))\n",
    "    \n",
    "    return inputs_real, inputs_z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generator network\n",
    "\n",
    "![GAN Network](assets/gan_network.png)\n",
    "\n",
    "Here we'll build the generator network. To make this network a universal function approximator, we'll need at least one hidden layer. We should use a leaky ReLU to allow gradients to flow backwards through the layer unimpeded. A leaky ReLU is like a normal ReLU, except that there is a small non-zero output for negative input values.\n",
    "\n",
    "#### Variable Scope\n",
    "Here we need to use `tf.variable_scope` for two reasons. Firstly, we're going to make sure all the variable names start with `generator`. Similarly, we'll prepend `discriminator` to the discriminator variables. This will help out later when we're training the separate networks.\n",
    "\n",
    "We could just use `tf.name_scope` to set the names, but we also want to reuse these networks with different inputs. For the generator, we're going to train it, but also _sample from it_ as we're training and after training. The discriminator will need to share variables between the fake and real input images. So, we can use the `reuse` keyword for `tf.variable_scope` to tell TensorFlow to reuse the variables instead of creating new ones if we build the graph again.\n",
    "\n",
    "To use `tf.variable_scope`, you use a `with` statement:\n",
    "```python\n",
    "with tf.variable_scope('scope_name', reuse=False):\n",
    "    # code here\n",
    "```\n",
    "\n",
    "Here's more from [the TensorFlow documentation](https://www.tensorflow.org/programmers_guide/variable_scope#the_problem) to get another look at using `tf.variable_scope`.\n",
    "\n",
    "#### Leaky ReLU\n",
    "TensorFlow doesn't provide an operation for leaky ReLUs, so we'll need to make one . For this you can use take the outputs from a linear fully connected layer and pass them to `tf.maximum`. Typically, a parameter `alpha` sets the magnitude of the output for negative values. So, the output for negative input (`x`) values is `alpha*x`, and the output for positive `x` is `x`:\n",
    "$$\n",
    "f(x) = max(\\alpha * x, x)\n",
    "$$\n",
    "\n",
    "#### Tanh Output\n",
    "The generator has been found to perform the best with $tanh$ for the generator output. This means that we'll have to rescale the MNIST images to be between -1 and 1, instead of 0 and 1.\n",
    "\n",
    ">**Exercise:** Implement the generator network in the function below. You'll need to return the tanh output. Make sure to wrap your code in a variable scope, with 'generator' as the scope name, and pass the `reuse` keyword argument from the function to `tf.variable_scope`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def generator(z, out_dim, n_units=128, reuse=False,  alpha=0.01):\n",
    "    ''' Build the generator network.\n",
    "    \n",
    "        Arguments\n",
    "        ---------\n",
    "        z : Input tensor for the generator\n",
    "        out_dim : Shape of the generator output\n",
    "        n_units : Number of units in hidden layer\n",
    "        reuse : Reuse the variables with tf.variable_scope\n",
    "        alpha : leak parameter for leaky ReLU\n",
    "        \n",
    "        Returns\n",
    "        -------\n",
    "        out, logits: \n",
    "    '''\n",
    "    with tf.variable_scope('generator', reuse=reuse):\n",
    "        # Hidden layer\n",
    "        zshape = z.get_shape().as_list()\n",
    "        print('zshape', zshape)\n",
    "        gw1 = tf.Variable(tf.random_normal([zshape[1], n_units], stddev=0.1), name=\"weight\")\n",
    "        gb1 = tf.Variable(tf.zeros([n_units]), name=\"bias\") \n",
    "        h1 = tf.matmul(z, gw1) + gb1\n",
    "        # Leaky ReLU\n",
    "        h1 = tf.maximum(alpha*h1, h1)\n",
    "        \n",
    "        # Logits and tanh output\n",
    "        gwo = tf.Variable(tf.random_normal([n_units, out_dim], stddev=0.1), name=\"output_weight\")\n",
    "        gbo = tf.Variable(tf.zeros([out_dim]), name=\"output_bias\")\n",
    "        logits = tf.matmul(h1, gwo) + gbo\n",
    "        out = tf.tanh(logits)\n",
    "        \n",
    "        return out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discriminator\n",
    "\n",
    "The discriminator network is almost exactly the same as the generator network, except that we're using a sigmoid output layer.\n",
    "\n",
    ">**Exercise:** Implement the discriminator network in the function below. Same as above, you'll need to return both the logits and the sigmoid output. Make sure to wrap your code in a variable scope, with 'discriminator' as the scope name, and pass the `reuse` keyword argument from the function arguments to `tf.variable_scope`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def discriminator(x, n_units=128, reuse=False, alpha=0.01):\n",
    "    ''' Build the discriminator network.\n",
    "    \n",
    "        Arguments\n",
    "        ---------\n",
    "        x : Input tensor for the discriminator\n",
    "        n_units: Number of units in hidden layer\n",
    "        reuse : Reuse the variables with tf.variable_scope\n",
    "        alpha : leak parameter for leaky ReLU\n",
    "        \n",
    "        Returns\n",
    "        -------\n",
    "        out, logits: \n",
    "    '''\n",
    "    with tf.variable_scope('discriminator', reuse=reuse):\n",
    "        # Hidden layer\n",
    "        xshape = x.get_shape().as_list()\n",
    "        print('xshape', xshape)\n",
    "        dw1 = tf.Variable(tf.random_normal([xshape[1], n_units], stddev=0.1), name=\"weight\")\n",
    "        db1 = tf.Variable(tf.zeros([n_units]), name=\"bias\") \n",
    "        h1 = tf.matmul(x, dw1) + db1\n",
    "        # Leaky ReLU\n",
    "        h1 = tf.maximum(alpha*h1, h1)\n",
    "        \n",
    "        dwo = tf.Variable(tf.random_normal([n_units, 1], stddev=0.1), name=\"output_weight\")\n",
    "        dbo = tf.Variable(tf.zeros([1]), name=\"output_bias\")\n",
    "        logits = tf.matmul(h1, dwo) + dbo\n",
    "        out = tf.sigmoid(logits)\n",
    "        \n",
    "        return out, logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Size of input image to discriminator\n",
    "input_size = 784 # 28x28 MNIST images flattened\n",
    "# Size of latent vector to generator\n",
    "z_size = 100\n",
    "# Sizes of hidden layers in generator and discriminator\n",
    "g_hidden_size = 128\n",
    "d_hidden_size = 128\n",
    "# Leak factor for leaky ReLU\n",
    "alpha = 0.01\n",
    "# Label smoothing \n",
    "smooth = 0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build network\n",
    "\n",
    "Now we're building the network from the functions defined above.\n",
    "\n",
    "First is to get our inputs, `input_real, input_z` from `model_inputs` using the sizes of the input and z.\n",
    "\n",
    "Then, we'll create the generator, `generator(input_z, input_size)`. This builds the generator with the appropriate input and output sizes.\n",
    "\n",
    "Then the discriminators. We'll build two of them, one for real data and one for fake data. Since we want the weights to be the same for both real and fake data, we need to reuse the variables. For the fake data, we're getting it from the generator as `g_model`. So the real data discriminator is `discriminator(input_real)` while the fake discriminator is `discriminator(g_model, reuse=True)`.\n",
    "\n",
    ">**Exercise:** Build the network from the functions you defined earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "zshape [None, 100]\n",
      "xshape [None, 784]\n",
      "xshape [None, 784]\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "# Create our input placeholders\n",
    "input_real, input_z = model_inputs(input_size, z_size)\n",
    "\n",
    "# Generator network here\n",
    "g_model = generator(input_z, input_size, g_hidden_size, False, alpha)\n",
    "# g_model is the generator output\n",
    "\n",
    "# Disriminator network here\n",
    "d_model_real, d_logits_real = discriminator(input_real, d_hidden_size, False, alpha)\n",
    "d_model_fake, d_logits_fake = discriminator(g_model, d_hidden_size, True, alpha)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discriminator and Generator Losses\n",
    "\n",
    "Now we need to calculate the losses, which is a little tricky. For the discriminator, the total loss is the sum of the losses for real and fake images, `d_loss = d_loss_real + d_loss_fake`. The losses will by sigmoid cross-entropys, which we can get with `tf.nn.sigmoid_cross_entropy_with_logits`. We'll also wrap that in `tf.reduce_mean` to get the mean for all the images in the batch. So the losses will look something like \n",
    "\n",
    "```python\n",
    "tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=logits, labels=labels))\n",
    "```\n",
    "\n",
    "For the real image logits, we'll use `d_logits_real` which we got from the discriminator in the cell above. For the labels, we want them to be all ones, since these are all real images. To help the discriminator generalize better, the labels are reduced a bit from 1.0 to 0.9, for example,  using the parameter `smooth`. This is known as label smoothing, typically used with classifiers to improve performance. In TensorFlow, it looks something like `labels = tf.ones_like(tensor) * (1 - smooth)`\n",
    "\n",
    "The discriminator loss for the fake data is similar. The logits are `d_logits_fake`, which we got from passing the generator output to the discriminator. These fake logits are used with labels of all zeros. Remember that we want the discriminator to output 1 for real images and 0 for fake images, so we need to set up the losses to reflect that.\n",
    "\n",
    "Finally, the generator losses are using `d_logits_fake`, the fake image logits. But, now the labels are all ones. The generator is trying to fool the discriminator, so it wants to discriminator to output ones for fake images.\n",
    "\n",
    ">**Exercise:** Calculate the losses for the discriminator and the generator. There are two discriminator losses, one for real images and one for fake images. For the real image loss, use the real logits and (smoothed) labels of ones. For the fake image loss, use the fake logits with labels of all zeros. The total discriminator loss is the sum of those two losses. Finally, the generator loss again uses the fake logits from the discriminator, but this time the labels are all ones because the generator wants to fool the discriminator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate losses\n",
    "labels_r = tf.ones_like(d_logits_real)*(1-smooth)\n",
    "d_loss_real = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=d_logits_real, labels=labels_r))\n",
    "\n",
    "labels_f = tf.zeros_like(d_logits_fake)\n",
    "d_loss_fake = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=d_logits_fake, labels=labels_f))\n",
    "\n",
    "d_loss = d_loss_real + d_loss_fake\n",
    "\n",
    "labels_g = tf.ones_like(d_logits_fake)\n",
    "g_loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=d_logits_fake, labels=labels_g))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimizers\n",
    "\n",
    "We want to update the generator and discriminator variables separately. So we need to get the variables for each part build optimizers for the two parts. To get all the trainable variables, we use `tf.trainable_variables()`. This creates a list of all the variables we've defined in our graph.\n",
    "\n",
    "For the generator optimizer, we only want to generator variables. Our past selves were nice and used a variable scope to start all of our generator variable names with `generator`. So, we just need to iterate through the list from `tf.trainable_variables()` and keep variables to start with `generator`. Each variable object has an attribute `name` which holds the name of the variable as a string (`var.name == 'weights_0'` for instance). \n",
    "\n",
    "We can do something similar with the discriminator. All the variables in the discriminator start with `discriminator`.\n",
    "\n",
    "Then, in the optimizer we pass the variable lists to `var_list` in the `minimize` method. This tells the optimizer to only update the listed variables. Something like `tf.train.AdamOptimizer().minimize(loss, var_list=var_list)` will only train the variables in `var_list`.\n",
    "\n",
    ">**Exercise: ** Below, implement the optimizers for the generator and discriminator. First you'll need to get a list of trainable variables, then split that list into two lists, one for the generator variables and another for the discriminator variables. Finally, using `AdamOptimizer`, create an optimizer for each network that update the network variables separately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Optimizers\n",
    "learning_rate = 0.002\n",
    "\n",
    "# Get the trainable_variables, split into G and D parts\n",
    "t_vars = tf.trainable_variables()\n",
    "g_vars = [v for v in t_vars if v.name.startswith(\"generator\")]\n",
    "d_vars = [v for v in t_vars if v.name.startswith(\"discriminator\")]\n",
    "\n",
    "d_train_opt = tf.train.AdamOptimizer().minimize(d_loss, var_list=d_vars)\n",
    "g_train_opt = tf.train.AdamOptimizer().minimize(g_loss, var_list=g_vars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100... Discriminator Loss: 0.3546... Generator Loss: 3.5729\n",
      "zshape [None, 100]\n"
     ]
    },
    {
     "ename": "FailedPreconditionError",
     "evalue": "Attempting to use uninitialized value generator_1/output_bias\n\t [[Node: generator_1/output_bias/read = Identity[T=DT_FLOAT, _class=[\"loc:@generator_1/output_bias\"], _device=\"/job:localhost/replica:0/task:0/cpu:0\"](generator_1/output_bias)]]\n\nCaused by op 'generator_1/output_bias/read', defined at:\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/runpy.py\", line 193, in _run_module_as_main\n    \"__main__\", mod_spec)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/ipykernel_launcher.py\", line 16, in <module>\n    app.launch_new_instance()\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/traitlets/config/application.py\", line 658, in launch_instance\n    app.start()\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/ipykernel/kernelapp.py\", line 477, in start\n    ioloop.IOLoop.instance().start()\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/zmq/eventloop/ioloop.py\", line 177, in start\n    super(ZMQIOLoop, self).start()\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/tornado/ioloop.py\", line 887, in start\n    handler_func(fd_obj, events)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/tornado/stack_context.py\", line 275, in null_wrapper\n    return fn(*args, **kwargs)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py\", line 440, in _handle_events\n    self._handle_recv()\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py\", line 472, in _handle_recv\n    self._run_callback(callback, msg)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py\", line 414, in _run_callback\n    callback(*args, **kwargs)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/tornado/stack_context.py\", line 275, in null_wrapper\n    return fn(*args, **kwargs)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/ipykernel/kernelbase.py\", line 283, in dispatcher\n    return self.dispatch_shell(stream, msg)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/ipykernel/kernelbase.py\", line 235, in dispatch_shell\n    handler(stream, idents, msg)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/ipykernel/kernelbase.py\", line 399, in execute_request\n    user_expressions, allow_stdin)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/ipykernel/ipkernel.py\", line 196, in do_execute\n    res = shell.run_cell(code, store_history=store_history, silent=silent)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/ipykernel/zmqshell.py\", line 533, in run_cell\n    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/IPython/core/interactiveshell.py\", line 2717, in run_cell\n    interactivity=interactivity, compiler=compiler, result=result)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/IPython/core/interactiveshell.py\", line 2821, in run_ast_nodes\n    if self.run_code(code, result):\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/IPython/core/interactiveshell.py\", line 2881, in run_code\n    exec(code_obj, self.user_global_ns, self.user_ns)\n  File \"<ipython-input-81-c104e1205ecd>\", line 36, in <module>\n    generator(input_z, input_size, reuse=True),\n  File \"<ipython-input-75-5e1b5689b082>\", line 28, in generator\n    gbo = tf.Variable(tf.zeros([out_dim]), name=\"output_bias\")\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/tensorflow/python/ops/variables.py\", line 226, in __init__\n    expected_shape=expected_shape)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/tensorflow/python/ops/variables.py\", line 344, in _init_from_args\n    self._snapshot = array_ops.identity(self._variable, name=\"read\")\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py\", line 1490, in identity\n    result = _op_def_lib.apply_op(\"Identity\", input=input, name=name)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py\", line 763, in apply_op\n    op_def=op_def)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/tensorflow/python/framework/ops.py\", line 2395, in create_op\n    original_op=self._default_original_op, op_def=op_def)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/tensorflow/python/framework/ops.py\", line 1264, in __init__\n    self._traceback = _extract_stack()\n\nFailedPreconditionError (see above for traceback): Attempting to use uninitialized value generator_1/output_bias\n\t [[Node: generator_1/output_bias/read = Identity[T=DT_FLOAT, _class=[\"loc:@generator_1/output_bias\"], _device=\"/job:localhost/replica:0/task:0/cpu:0\"](generator_1/output_bias)]]\n",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mFailedPreconditionError\u001b[0m                   Traceback (most recent call last)",
      "\u001b[0;32m/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m   1021\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1022\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1023\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOpError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run_fn\u001b[0;34m(session, feed_dict, fetch_list, target_list, options, run_metadata)\u001b[0m\n\u001b[1;32m   1003\u001b[0m                                  \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget_list\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1004\u001b[0;31m                                  status, run_metadata)\n\u001b[0m\u001b[1;32m   1005\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/hernan/anaconda/envs/udcttf/lib/python3.6/contextlib.py\u001b[0m in \u001b[0;36m__exit__\u001b[0;34m(self, type, value, traceback)\u001b[0m\n\u001b[1;32m     88\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 89\u001b[0;31m                 \u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgen\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     90\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mStopIteration\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py\u001b[0m in \u001b[0;36mraise_exception_on_not_ok_status\u001b[0;34m()\u001b[0m\n\u001b[1;32m    468\u001b[0m           \u001b[0mcompat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpywrap_tensorflow\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_Message\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstatus\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 469\u001b[0;31m           pywrap_tensorflow.TF_GetCode(status))\n\u001b[0m\u001b[1;32m    470\u001b[0m   \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mFailedPreconditionError\u001b[0m: Attempting to use uninitialized value generator_1/output_bias\n\t [[Node: generator_1/output_bias/read = Identity[T=DT_FLOAT, _class=[\"loc:@generator_1/output_bias\"], _device=\"/job:localhost/replica:0/task:0/cpu:0\"](generator_1/output_bias)]]",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mFailedPreconditionError\u001b[0m                   Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-81-c104e1205ecd>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     35\u001b[0m         gen_samples = sess.run(\n\u001b[1;32m     36\u001b[0m                        \u001b[0mgenerator\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput_z\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreuse\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 37\u001b[0;31m                        feed_dict={input_z: sample_z})\n\u001b[0m\u001b[1;32m     38\u001b[0m         \u001b[0msamples\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgen_samples\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     39\u001b[0m         \u001b[0msaver\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msave\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msess\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'./checkpoints/generator.ckpt'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    765\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    766\u001b[0m       result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 767\u001b[0;31m                          run_metadata_ptr)\n\u001b[0m\u001b[1;32m    768\u001b[0m       \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    769\u001b[0m         \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    963\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mfinal_fetches\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mfinal_targets\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    964\u001b[0m       results = self._do_run(handle, final_targets, final_fetches,\n\u001b[0;32m--> 965\u001b[0;31m                              feed_dict_string, options, run_metadata)\n\u001b[0m\u001b[1;32m    966\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    967\u001b[0m       \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_run\u001b[0;34m(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1013\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1014\u001b[0m       return self._do_call(_run_fn, self._session, feed_dict, fetch_list,\n\u001b[0;32m-> 1015\u001b[0;31m                            target_list, options, run_metadata)\n\u001b[0m\u001b[1;32m   1016\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1017\u001b[0m       return self._do_call(_prun_fn, self._session, handle, feed_dict,\n",
      "\u001b[0;32m/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m   1033\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1034\u001b[0m           \u001b[0;32mpass\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1035\u001b[0;31m       \u001b[0;32mraise\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnode_def\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mop\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1036\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1037\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_extend_graph\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mFailedPreconditionError\u001b[0m: Attempting to use uninitialized value generator_1/output_bias\n\t [[Node: generator_1/output_bias/read = Identity[T=DT_FLOAT, _class=[\"loc:@generator_1/output_bias\"], _device=\"/job:localhost/replica:0/task:0/cpu:0\"](generator_1/output_bias)]]\n\nCaused by op 'generator_1/output_bias/read', defined at:\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/runpy.py\", line 193, in _run_module_as_main\n    \"__main__\", mod_spec)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/ipykernel_launcher.py\", line 16, in <module>\n    app.launch_new_instance()\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/traitlets/config/application.py\", line 658, in launch_instance\n    app.start()\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/ipykernel/kernelapp.py\", line 477, in start\n    ioloop.IOLoop.instance().start()\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/zmq/eventloop/ioloop.py\", line 177, in start\n    super(ZMQIOLoop, self).start()\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/tornado/ioloop.py\", line 887, in start\n    handler_func(fd_obj, events)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/tornado/stack_context.py\", line 275, in null_wrapper\n    return fn(*args, **kwargs)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py\", line 440, in _handle_events\n    self._handle_recv()\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py\", line 472, in _handle_recv\n    self._run_callback(callback, msg)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py\", line 414, in _run_callback\n    callback(*args, **kwargs)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/tornado/stack_context.py\", line 275, in null_wrapper\n    return fn(*args, **kwargs)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/ipykernel/kernelbase.py\", line 283, in dispatcher\n    return self.dispatch_shell(stream, msg)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/ipykernel/kernelbase.py\", line 235, in dispatch_shell\n    handler(stream, idents, msg)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/ipykernel/kernelbase.py\", line 399, in execute_request\n    user_expressions, allow_stdin)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/ipykernel/ipkernel.py\", line 196, in do_execute\n    res = shell.run_cell(code, store_history=store_history, silent=silent)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/ipykernel/zmqshell.py\", line 533, in run_cell\n    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/IPython/core/interactiveshell.py\", line 2717, in run_cell\n    interactivity=interactivity, compiler=compiler, result=result)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/IPython/core/interactiveshell.py\", line 2821, in run_ast_nodes\n    if self.run_code(code, result):\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/IPython/core/interactiveshell.py\", line 2881, in run_code\n    exec(code_obj, self.user_global_ns, self.user_ns)\n  File \"<ipython-input-81-c104e1205ecd>\", line 36, in <module>\n    generator(input_z, input_size, reuse=True),\n  File \"<ipython-input-75-5e1b5689b082>\", line 28, in generator\n    gbo = tf.Variable(tf.zeros([out_dim]), name=\"output_bias\")\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/tensorflow/python/ops/variables.py\", line 226, in __init__\n    expected_shape=expected_shape)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/tensorflow/python/ops/variables.py\", line 344, in _init_from_args\n    self._snapshot = array_ops.identity(self._variable, name=\"read\")\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py\", line 1490, in identity\n    result = _op_def_lib.apply_op(\"Identity\", input=input, name=name)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py\", line 763, in apply_op\n    op_def=op_def)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/tensorflow/python/framework/ops.py\", line 2395, in create_op\n    original_op=self._default_original_op, op_def=op_def)\n  File \"/Users/hernan/anaconda/envs/udcttf/lib/python3.6/site-packages/tensorflow/python/framework/ops.py\", line 1264, in __init__\n    self._traceback = _extract_stack()\n\nFailedPreconditionError (see above for traceback): Attempting to use uninitialized value generator_1/output_bias\n\t [[Node: generator_1/output_bias/read = Identity[T=DT_FLOAT, _class=[\"loc:@generator_1/output_bias\"], _device=\"/job:localhost/replica:0/task:0/cpu:0\"](generator_1/output_bias)]]\n"
     ]
    }
   ],
   "source": [
    "batch_size = 100\n",
    "epochs = 100\n",
{'fraction_non_alphanumeric': 0.14839101613450784, 'fraction_numerical': 0.054389193631961694, 'mean_word_length': 3.3810567514677103, 'pattern_counts': {'":': 192, '<': 15, '<?xml version=': 0, '>': 36, 'https://': 10, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4013459', 'n_tokens_mistral': 22323, 'n_tokens_neox': 20441, 'n_words': 4849}