<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Session 3: Unsupervised and Supervised Learning\n",
    "\n",
    "<p class=\"lead\">\n",
    "Assignment: Build Unsupervised and Supervised Networks\n",
    "</p>\n",
    "\n",
    "<p class=\"lead\">\n",
    "<NAME><br />\n",
    "<a href=\"https://www.kadenze.com/courses/creative-applications-of-deep-learning-with-tensorflow/info\">Creative Applications of Deep Learning w/ Tensorflow</a><br />\n",
    "<a href=\"https://www.kadenze.com/partners/kadenze-academy\">Kadenze Academy</a><br />\n",
    "<a href=\"https://twitter.com/hashtag/CADL\">#CADL</a>\n",
    "</p>\n",
    "\n",
    "<a name=\"learning-goals\"></a>\n",
    "# Learning Goals\n",
    "\n",
    "* Learn how to build an autoencoder\n",
    "* Learn how to explore latent/hidden representations of an autoencoder.\n",
    "* Learn how to build a classification network using softmax and onehot encoding\n",
    "\n",
    "# Outline\n",
    "\n",
    "<!-- MarkdownTOC autolink=true autoanchor=true bracket=round -->\n",
    "\n",
    "- [Assignment Synopsis](#assignment-synopsis)\n",
    "- [Part One - Autoencoders](#part-one---autoencoders)\n",
    "    - [Instructions](#instructions)\n",
    "    - [Code](#code)\n",
    "    - [Visualize the Embedding](#visualize-the-embedding)\n",
    "    - [Reorganize to Grid](#reorganize-to-grid)\n",
    "    - [2D Latent Manifold](#2d-latent-manifold)\n",
    "- [Part Two - General Autoencoder Framework](#part-two---general-autoencoder-framework)\n",
    "    - [Instructions](#instructions-1)\n",
    "- [Part Three - Deep Audio Classification Network](#part-three---deep-audio-classification-network)\n",
    "    - [Instructions](#instructions-2)\n",
    "    - [Preparing the Data](#preparing-the-data)\n",
    "    - [Creating the Network](#creating-the-network)\n",
    "- [Assignment Submission](#assignment-submission)\n",
    "- [Coming Up](#coming-up)\n",
    "\n",
    "<!-- /MarkdownTOC -->\n",
    "\n",
    "This next section will just make sure you have the right version of python and the libraries that we'll be using.  Don't change the code here but make sure you \"run\" it (use \"shift+enter\")!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# First check the Python version\n",
    "import sys\n",
    "if sys.version_info < (3,4):\n",
    "    print('You are running an older version of Python!\\n\\n' \\\n",
    "          'You should consider updating to Python 3.4.0 or ' \\\n",
    "          'higher as the libraries built for this course ' \\\n",
    "          'have only been tested in Python 3.4 and higher.\\n')\n",
    "    print('Try installing the Python 3.5 version of anaconda '\n",
    "          'and then restart `jupyter notebook`:\\n' \\\n",
    "          'https://www.continuum.io/downloads\\n\\n')\n",
    "\n",
    "# Now get necessary libraries\n",
    "try:\n",
    "    import os\n",
    "    import numpy as np\n",
    "    import matplotlib.pyplot as plt\n",
    "    from skimage.transform import resize\n",
    "    from skimage import data\n",
    "    from scipy.misc import imresize\n",
    "    import IPython.display as ipyd\n",
    "except ImportError:\n",
    "    print('You are missing some packages! ' \\\n",
    "          'We will try installing them before continuing!')\n",
    "    !pip install \"numpy>=1.11.0\" \"matplotlib>=1.5.1\" \"scikit-image>=0.11.3\" \"scikit-learn>=0.17\" \"scipy>=0.17.0\"\n",
    "    import os\n",
    "    import numpy as np\n",
    "    import matplotlib.pyplot as plt\n",
    "    from skimage.transform import resize\n",
    "    from skimage import data\n",
    "    from scipy.misc import imresize\n",
    "    import IPython.display as ipyd\n",
    "    print('Done!')\n",
    "\n",
    "# Import Tensorflow\n",
    "try:\n",
    "    import tensorflow as tf\n",
    "except ImportError:\n",
    "    print(\"You do not have tensorflow installed!\")\n",
    "    print(\"Follow the instructions on the following link\")\n",
    "    print(\"to install tensorflow before continuing:\")\n",
    "    print(\"\")\n",
    "    print(\"https://github.com/pkmital/CADL#installation-preliminaries\")\n",
    "\n",
    "# This cell includes the provided libraries from the zip file\n",
    "# and a library for displaying images from ipython, which\n",
    "# we will use to display the gif\n",
    "try:\n",
    "    from libs import utils, gif, datasets, dataset_utils, vae, dft\n",
    "except ImportError:\n",
    "    print(\"Make sure you have started notebook in the same directory\" +\n",
    "          \" as the provided zip file which includes the 'libs' folder\" +\n",
    "          \" and the file 'utils.py' inside of it.  You will NOT be able\"\n",
    "          \" to complete this assignment unless you restart jupyter\"\n",
    "          \" notebook inside the directory created by extracting\"\n",
    "          \" the zip file or cloning the github repo.\")\n",
    "\n",
    "# We'll tell matplotlib to inline any drawn figures like so:\n",
    "%matplotlib inline\n",
    "plt.style.use('ggplot')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Bit of formatting because I don't like the default inline code style:\n",
    "from IPython.core.display import HTML\n",
    "HTML(\"\"\"<style> .rendered_html code { \n",
    "    padding: 2px 4px;\n",
    "    color: #c7254e;\n",
    "    background-color: #f9f2f4;\n",
    "    border-radius: 4px;\n",
    "} </style>\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"assignment-synopsis\"></a>\n",
    "# Assignment Synopsis\n",
    "\n",
    "In the last session we created our first neural network.  We saw that in order to create a neural network, we needed to define a cost function which would allow gradient descent to optimize all the parameters in our network.  We also saw how neural networks become much more expressive by introducing series of linearities followed by non-linearities, or activation functions.  We then explored a fun application of neural networks using regression to learn to paint color values given x, y positions.  This allowed us to build up a sort of painterly like version of an image.\n",
    "\n",
    "In this session, we'll see how to construct a few more types of neural networks.  First, we'll explore a generative network called autoencoders.  This network can be extended in a variety of ways to include convolution, denoising, or a variational layer.  In Part Two, you'll then use a general autoencoder framework to encode your own list of images.  In Part three, we'll then explore a discriminative network used for classification, and see how this can be used for audio classification of music or speech.\n",
    "\n",
    "One main difference between these two networks are the data that we'll use to train them.  In the first case, we will only work with \"unlabeled\" data and perform unsupervised learning.  An example would be a collection of images, just like the one you created for assignment 1.  Contrast this with \"labeled\" data which allows us to make use of supervised learning.  For instance, we're given both images, and some other data about those images such as some text describing what object is in the image.  This allows us to optimize a network where we model a distribution over the images given that it should be labeled as something.  This is often a *much* simpler distribution to train, but with the expense of it being much harder to collect.\n",
    "\n",
    "One of the major directions of future research will be in how to better make use of unlabeled data and unsupervised learning methods.\n",
    "\n",
    "<a name=\"part-one---autoencoders\"></a>\n",
    "# Part One - Autoencoders\n",
    "\n",
    "<a name=\"instructions\"></a>\n",
    "## Instructions\n",
    "\n",
    "Work with a dataset of images and train an autoencoder.  You can work with the same dataset from assignment 1, or try a larger dataset.  But be careful with the image sizes, and make sure to keep it relatively small (e.g. < 200 x 200 px).  \n",
    "\n",
    "Recall from the lecture that autoencoders are great at \"compressing\" information.  The network's construction and cost function are just like what we've done in the last session.  The network is composed of a series of matrix multiplications and nonlinearities.  The only difference is the output of the network has exactly the same shape as what is input.  This allows us to train the network by saying that the output of the network needs to be just like the input to it, so that it tries to \"compress\" all the information in that video.\n",
    "\n",
    "Autoencoders have some great potential for creative applications, as they allow us to compress a dataset of information and even *generate* new data from that encoding.  We'll see exactly how to do this with a basic autoencoder, and then you'll be asked to explore some of the extensions to produce your own encodings.\n",
    "\n",
    "<a name=\"code\"></a>\n",
    "## Code\n",
    "\n",
    "We'll now go through the process of building an autoencoder just like in the lecture.  First, let's load some data.  You can use the first 100 images of the Celeb Net, your own dataset, or anything else approximately under 1,000 images.  Make sure you resize the images so that they are <= 200x200 pixels, otherwise the training will be *very* slow, and the montages we create will be too large.\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# See how this works w/ Celeb Images or try your own dataset instead:\n",
    "imgs = ...\n",
    "\n",
    "# Then convert the list of images to a 4d array (e.g. use np.array to convert a list to a 4d array):\n",
    "Xs = ...\n",
    "\n",
    "print(Xs.shape)\n",
    "assert(Xs.ndim == 4 and Xs.shape[1] <= 250 and Xs.shape[2] <= 250)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll now make use of something I've written to help us store this data.  It provides some interfaces for generating \"batches\" of data, as well as splitting the data into training, validation, and testing sets.  To use it, we pass in the data and optionally its labels.  If we don't have labels, we just pass in the data.  In the second half of this notebook, we'll explore using a dataset's labels as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ds = datasets.Dataset(Xs)\n",
    "# ds = datasets.CIFAR10(flatten=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It allows us to easily find the mean:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "mean_img = ds.mean().astype(np.uint8)\n",
    "plt.imshow(mean_img)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or the deviation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "std_img = ds.std().astype(np.uint8)\n",
    "plt.imshow(std_img)\n",
    "print(std_img.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall we can calculate the mean of the standard deviation across each color channel:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "std_img = np.mean(std_img, axis=2).astype(np.uint8)\n",
    "plt.imshow(std_img)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the input data we gave as input to our `Datasets` object, previously stored in `Xs` is now stored in a variable as part of our `ds` Datasets object, `X`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.imshow(ds.X[0])\n",
    "print(ds.X.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It takes a parameter, `split` at the time of creation, which allows us to create train/valid/test sets.  By default, this is set to `[1.0, 0.0, 0.0]`, which means to take all the data in the train set, and nothing in the validation and testing sets.  We can access \"batch generators\" of each of these sets by saying: `ds.train.next_batch`.  A generator is a really powerful way of handling iteration in Python.  If you are unfamiliar with the idea of generators, I recommend reading up a little bit on it, e.g. here: http://intermediatepythonista.com/python-generators - think of it as a for loop, but as a function.  It returns one iteration of the loop each time you call it.\n",
    "\n",
    "This generator will automatically handle the randomization of the dataset.  Let's try looping over the dataset using the batch generator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for (X, y) in ds.train.next_batch(batch_size=10):\n",
    "    print(X.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This returns `X` and `y` as a tuple.  Since we're not using labels, we'll just ignore this.  The `next_batch` method takes a parameter, `batch_size`, which we'll set appropriately to our batch size.  Notice it runs for exactly 10 iterations to iterate over our 100 examples, then the loop exits.  The order in which it iterates over the 100 examples is randomized each time you iterate.\n",
    "\n",
    "Write two functions to preprocess (normalize) any given image, and to unprocess it, i.e. unnormalize it by removing the normalization.  The `preprocess` function should perform exactly the task you learned to do in assignment 1: subtract the mean, then divide by the standard deviation.  The `deprocess` function should take the preprocessed image and undo the preprocessing steps.  Recall that the `ds` object contains the `mean` and `std` functions for access the mean and standarad deviation.  We'll be using the `preprocess` and `deprocess` functions on the input and outputs of the network.  Note, we could use Tensorflow to do this instead of numpy, but for sake of clarity, I'm keeping this separate from the Tensorflow graph.\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Write a function to preprocess/normalize an image, given its dataset object\n",
    "# (which stores the mean and standard deviation!)\n",
    "def preprocess(img, ds):\n",
    "    norm_img = (img - ...) / ...\n",
    "    return norm_img\n",
    "\n",
    "# Write a function to undo the normalization of an image, given its dataset object\n",
    "# (which stores the mean and standard deviation!)\n",
    "def deprocess(norm_img, ds):\n",
    "    img = norm_img * ... + ...\n",
    "    return img"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're going to now work on creating an autoencoder.  To start, we'll only use linear connections, like in the last assignment.  This means, we need a 2-dimensional input:  Batch Size x Number of Features.  We currently have a 4-dimensional input: Batch Size x Height x Width x Channels.  We'll have to calculate the number of features we have to help construct the Tensorflow Graph for our autoencoder neural network.  Then, when we are ready to train the network, we'll reshape our 4-dimensional dataset into a 2-dimensional one when feeding the input of the network.  Optionally, we could create a `tf.reshape` as the first operation of the network, so that we can still pass in our 4-dimensional array, and the Tensorflow graph would reshape it for us.  We'll try the former method, by reshaping manually, and then you can explore the latter method, of handling 4-dimensional inputs on your own.\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Calculate the number of features in your image.\n",
    "# This is the total number of pixels, or (height x width x channels).\n",
    "n_features = ...\n",
    "print(n_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a list of how many neurons we want in each layer.  This should be for just one half of the network, the encoder only.  It should start large, then get smaller and smaller.  We're also going to try an encode our dataset to an inner layer of just 2 values.  So from our number of features, we'll go all the way down to expressing that image by just 2 values.  Try the values I've put here for the celeb dataset, then explore your own values:\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "encoder_dimensions = [2048, 512, 128, 2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now create a placeholder just like in the last session in the tensorflow graph that will be able to get any number (None) of `n_features` inputs.\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X = tf.placeholder(...\n",
    "                   \n",
    "assert(X.get_shape().as_list() == [None, n_features])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now complete the function `encode` below.  This takes as input our input placeholder, `X`, our list of `dimensions`, and an `activation` function, e.g. `tf.nn.relu` or `tf.nn.tanh`, to apply to each layer's output, and creates a series of fully connected layers.  This works just like in the last session!  We multiply our input, add a bias, then apply a non-linearity.  Instead of having 20 neurons in each layer, we're going to use our `dimensions` list to tell us how many neurons we want in each layer.\n",
    "\n",
    "One important difference is that we're going to also store every weight matrix we create!  This is so that we can use the same weight matrices when we go to build our decoder.  This is a *very* powerful concept that creeps up in a few different neural network architectures called weight sharing.  Weight sharing isn't necessary to do of course, but can speed up training and offer a different set of features depending on your dataset.  Explore trying both.  We'll also see how another form of weight sharing works in convolutional networks.\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def encode(X, dimensions, activation=tf.nn.tanh):\n",
    "    # We're going to keep every matrix we create so let's create a list to hold them all\n",
    "    Ws = []\n",
    "\n",
    "    # We'll create a for loop to create each layer:\n",
    "    for layer_i, n_output in enumerate(dimensions):\n",
    "\n",
    "        # TODO: just like in the last session,\n",
    "        # we'll use a variable scope to help encapsulate our variables\n",
    "        # This will simply prefix all the variables made in this scope\n",
    "        # with the name we give it.  Make sure it is a unique name\n",
    "        # for each layer, e.g., 'encoder/layer1', 'encoder/layer2', or\n",
    "        # 'encoder/1', 'encoder/2',... \n",
    "        with tf.variable_scope(...)\n",
    "\n",
    "            # TODO: Create a weight matrix which will increasingly reduce\n",
    "            # down the amount of information in the input by performing\n",
    "            # a matrix multiplication.  You can use the utils.linear function.\n",
    "            h, W = ...\n",
    "\n",
    "            # Finally we'll store the weight matrix.\n",
    "            # We need to keep track of all\n",
    "            # the weight matrices we've used in our encoder\n",
    "            # so that we can build the decoder using the\n",
    "            # same weight matrices.\n",
    "            Ws.append(W)\n",
    "            \n",
    "            # Replace X with the current layer's output, so we can\n",
    "            # use it in the next layer.\n",
    "            X = h\n",
    "    \n",
    "    z = X\n",
    "    return Ws, z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Then call the function\n",
    "Ws, z = encode(X, encoder_dimensions)\n",
    "\n",
    "# And just some checks to make sure you've done it right.\n",
    "assert(z.get_shape().as_list() == [None, 2])\n",
    "assert(len(Ws) == len(encoder_dimensions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the graph:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "[op.name for op in tf.get_default_graph().get_operations()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we've created a few layers, encoding our input `X` all the way down to 2 values in the tensor `z`.  We do this by multiplying our input `X` by a set of matrices shaped as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "[W_i.get_shape().as_list() for W_i in Ws]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Resulting in a layer which is shaped as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "z.get_shape().as_list()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the Decoder \n",
    "\n",
    "Here is a helpful animation on what the matrix \"transpose\" operation does:\n",
    "![transpose](https://upload.wikimedia.org/wikipedia/commons/e/e4/Matrix_transpose.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basically what is happening is rows becomes columns, and vice-versa.  We're going to use our existing weight matrices but transpose them so that we can go in the opposite direction.  In order to build our decoder, we'll have to do the opposite of what we've just done, multiplying `z` by the transpose of our weight matrices, to get back to a reconstructed version of `X`.  First, we'll reverse the order of our weight matrics, and then append to the list of dimensions the final output layer's shape to match our input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# We'll first reverse the order of our weight matrices\n",
    "decoder_Ws = Ws[::-1]\n",
    "\n",
    "# then reverse the order of our dimensions\n",
    "# appending the last layers number of inputs.\n",
    "decoder_dimensions = encoder_dimensions[::-1][1:] + [n_features]\n",
    "print(decoder_dimensions)\n",
    "\n",
    "assert(decoder_dimensions[-1] == n_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll build the decoder.  I've shown you how to do this.  Read through the code to fully understand what it is doing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def decode(z, dimensions, Ws, activation=tf.nn.tanh):\n",
    "    current_input = z\n",
    "    for layer_i, n_output in enumerate(dimensions):\n",
    "        # we'll use a variable scope again to help encapsulate our variables\n",
    "        # This will simply prefix all the variables made in this scope\n",
    "        # with the name we give it.\n",
    "        with tf.variable_scope(\"decoder/layer/{}\".format(layer_i)):\n",
    "\n",
    "            # Now we'll grab the weight matrix we created before and transpose it\n",
    "            # So a 3072 x 784 matrix would become 784 x 3072\n",
    "            # or a 256 x 64 matrix, would become 64 x 256\n",
    "            W = tf.transpose(Ws[layer_i])\n",
    "\n",
    "            # Now we'll multiply our input by our transposed W matrix\n",
    "            h = tf.matmul(current_input, W)\n",
    "\n",
    "            # And then use a relu activation function on its output\n",
    "            current_input = activation(h)\n",
    "\n",
    "            # We'll also replace n_input with the current n_output, so that on the\n",
    "            # next iteration, our new number inputs will be correct.\n",
    "            n_input = n_output\n",
    "    Y = current_input\n",
    "    return Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Y = decode(z, decoder_dimensions, decoder_Ws)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the new operations we've just added.  They will all be prefixed by \"decoder\" so we can use list comprehension to help us with this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "[op.name for op in tf.get_default_graph().get_operations()\n",
    " if op.name.startswith('decoder')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's take a look at the output of the autoencoder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Y.get_shape().as_list()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great!  So we should have a synthesized version of our input placeholder, `X`, inside of `Y`.  This `Y` is the result of many matrix multiplications, first a series of multiplications in our encoder all the way down to 2 dimensions, and then back to the original dimensions through our decoder.  Let's now create a pixel-to-pixel measure of error.  This should measure the difference in our synthesized output, `Y`, and our input, `X`.  You can use the $l_1$ or $l_2$ norm, just like in assignment 2.  If you don't remember, go back to homework 2 where we calculated the cost function and try the same idea here.\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Calculate some measure of loss, e.g. the pixel to pixel absolute difference or squared difference\n",
    "loss = ...\n",
    "\n",
    "# Now sum over every pixel and then calculate the mean over the batch dimension (just like session 2!)\n",
    "# hint, use tf.reduce_mean and tf.reduce_sum\n",
    "cost = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now for the standard training code.  We'll pass our `cost` to an optimizer, and then use mini batch gradient descent to optimize our network's parameters.  We just have to be careful to make sure we're preprocessing our input and feed it in the right shape, a 2-dimensional matrix of [batch_size, n_features] in dimensions.\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learning_rate = ...\n",
    "optimizer = tf.train.AdamOptimizer(...).minimize(...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is the training code for our autoencoder.  Please go through each line of code to make sure you understand what is happening, and fill in the missing pieces.  This will take awhile.  On my machine, it takes about 15 minutes.  If you're impatient, you can \"Interrupt\" the kernel by going to the Kernel menu above, and continue with the notebook.  Though, the longer you leave this to train, the better the result will be.\n",
    "\n",
    "What I really want you to notice is what the network learns to encode first, based on what it is able to reconstruct.  It won't able to reconstruct everything.  At first, it will just be the mean image.  Then, other major changes in the dataset.  For the first 100 images of celeb net, this seems to be the background: white, blue, black backgrounds.  From this basic interpretation, you can reason that the autoencoder has learned a representation of the backgrounds, and is able to encode that knowledge of the background in its inner most layer of just two values.  It then goes on to represent the major variations in skin tone and hair.  Then perhaps some facial features such as lips.  So the features it is able to encode tend to be the major things at first, then the smaller things.\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# (TODO) Create a tensorflow session and initialize all of our weights:\n",
    "sess = ...\n",
    "sess.run(tf.global_variables_initializer())\n",
    "\n",
    "# Some parameters for training\n",
    "batch_size = 100\n",
    "n_epochs = 31\n",
    "step = 10\n",
    "\n",
    "# We'll try to reconstruct the same first 100 images and show how\n",
    "# The network does over the course of training.\n",
    "examples = ds.X[:100]\n",<commit_msg>adding note about potential memory issues<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Session 3: Unsupervised and Supervised Learning\n",
    "\n",
    "<p class=\"lead\">\n",
    "Assignment: Build Unsupervised and Supervised Networks\n",
    "</p>\n",
    "\n",
    "<p class=\"lead\">\n",
    "<NAME><br />\n",
    "<a href=\"https://www.kadenze.com/courses/creative-applications-of-deep-learning-with-tensorflow/info\">Creative Applications of Deep Learning w/ Tensorflow</a><br />\n",
    "<a href=\"https://www.kadenze.com/partners/kadenze-academy\">Kadenze Academy</a><br />\n",
    "<a href=\"https://twitter.com/hashtag/CADL\">#CADL</a>\n",
    "</p>\n",
    "\n",
    "<a name=\"learning-goals\"></a>\n",
    "# Learning Goals\n",
    "\n",
    "* Learn how to build an autoencoder\n",
    "* Learn how to explore latent/hidden representations of an autoencoder.\n",
    "* Learn how to build a classification network using softmax and onehot encoding\n",
    "\n",
    "# Outline\n",
    "\n",
    "<!-- MarkdownTOC autolink=true autoanchor=true bracket=round -->\n",
    "\n",
    "- [Assignment Synopsis](#assignment-synopsis)\n",
    "- [Part One - Autoencoders](#part-one---autoencoders)\n",
    "    - [Instructions](#instructions)\n",
    "    - [Code](#code)\n",
    "    - [Visualize the Embedding](#visualize-the-embedding)\n",
    "    - [Reorganize to Grid](#reorganize-to-grid)\n",
    "    - [2D Latent Manifold](#2d-latent-manifold)\n",
    "- [Part Two - General Autoencoder Framework](#part-two---general-autoencoder-framework)\n",
    "    - [Instructions](#instructions-1)\n",
    "- [Part Three - Deep Audio Classification Network](#part-three---deep-audio-classification-network)\n",
    "    - [Instructions](#instructions-2)\n",
    "    - [Preparing the Data](#preparing-the-data)\n",
    "    - [Creating the Network](#creating-the-network)\n",
    "- [Assignment Submission](#assignment-submission)\n",
    "- [Coming Up](#coming-up)\n",
    "\n",
    "<!-- /MarkdownTOC -->\n",
    "\n",
    "This next section will just make sure you have the right version of python and the libraries that we'll be using.  Don't change the code here but make sure you \"run\" it (use \"shift+enter\")!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# First check the Python version\n",
    "import sys\n",
    "if sys.version_info < (3,4):\n",
    "    print('You are running an older version of Python!\\n\\n' \\\n",
    "          'You should consider updating to Python 3.4.0 or ' \\\n",
    "          'higher as the libraries built for this course ' \\\n",
    "          'have only been tested in Python 3.4 and higher.\\n')\n",
    "    print('Try installing the Python 3.5 version of anaconda '\n",
    "          'and then restart `jupyter notebook`:\\n' \\\n",
    "          'https://www.continuum.io/downloads\\n\\n')\n",
    "\n",
    "# Now get necessary libraries\n",
    "try:\n",
    "    import os\n",
    "    import numpy as np\n",
    "    import matplotlib.pyplot as plt\n",
    "    from skimage.transform import resize\n",
    "    from skimage import data\n",
    "    from scipy.misc import imresize\n",
    "    import IPython.display as ipyd\n",
    "except ImportError:\n",
    "    print('You are missing some packages! ' \\\n",
    "          'We will try installing them before continuing!')\n",
    "    !pip install \"numpy>=1.11.0\" \"matplotlib>=1.5.1\" \"scikit-image>=0.11.3\" \"scikit-learn>=0.17\" \"scipy>=0.17.0\"\n",
    "    import os\n",
    "    import numpy as np\n",
    "    import matplotlib.pyplot as plt\n",
    "    from skimage.transform import resize\n",
    "    from skimage import data\n",
    "    from scipy.misc import imresize\n",
    "    import IPython.display as ipyd\n",
    "    print('Done!')\n",
    "\n",
    "# Import Tensorflow\n",
    "try:\n",
    "    import tensorflow as tf\n",
    "except ImportError:\n",
    "    print(\"You do not have tensorflow installed!\")\n",
    "    print(\"Follow the instructions on the following link\")\n",
    "    print(\"to install tensorflow before continuing:\")\n",
    "    print(\"\")\n",
    "    print(\"https://github.com/pkmital/CADL#installation-preliminaries\")\n",
    "\n",
    "# This cell includes the provided libraries from the zip file\n",
    "# and a library for displaying images from ipython, which\n",
    "# we will use to display the gif\n",
    "try:\n",
    "    from libs import utils, gif, datasets, dataset_utils, vae, dft\n",
    "except ImportError:\n",
    "    print(\"Make sure you have started notebook in the same directory\" +\n",
    "          \" as the provided zip file which includes the 'libs' folder\" +\n",
    "          \" and the file 'utils.py' inside of it.  You will NOT be able\"\n",
    "          \" to complete this assignment unless you restart jupyter\"\n",
    "          \" notebook inside the directory created by extracting\"\n",
    "          \" the zip file or cloning the github repo.\")\n",
    "\n",
    "# We'll tell matplotlib to inline any drawn figures like so:\n",
    "%matplotlib inline\n",
    "plt.style.use('ggplot')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Bit of formatting because I don't like the default inline code style:\n",
    "from IPython.core.display import HTML\n",
    "HTML(\"\"\"<style> .rendered_html code { \n",
    "    padding: 2px 4px;\n",
    "    color: #c7254e;\n",
    "    background-color: #f9f2f4;\n",
    "    border-radius: 4px;\n",
    "} </style>\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"assignment-synopsis\"></a>\n",
    "# Assignment Synopsis\n",
    "\n",
    "In the last session we created our first neural network.  We saw that in order to create a neural network, we needed to define a cost function which would allow gradient descent to optimize all the parameters in our network.  We also saw how neural networks become much more expressive by introducing series of linearities followed by non-linearities, or activation functions.  We then explored a fun application of neural networks using regression to learn to paint color values given x, y positions.  This allowed us to build up a sort of painterly like version of an image.\n",
    "\n",
    "In this session, we'll see how to construct a few more types of neural networks.  First, we'll explore a generative network called autoencoders.  This network can be extended in a variety of ways to include convolution, denoising, or a variational layer.  In Part Two, you'll then use a general autoencoder framework to encode your own list of images.  In Part three, we'll then explore a discriminative network used for classification, and see how this can be used for audio classification of music or speech.\n",
    "\n",
    "One main difference between these two networks are the data that we'll use to train them.  In the first case, we will only work with \"unlabeled\" data and perform unsupervised learning.  An example would be a collection of images, just like the one you created for assignment 1.  Contrast this with \"labeled\" data which allows us to make use of supervised learning.  For instance, we're given both images, and some other data about those images such as some text describing what object is in the image.  This allows us to optimize a network where we model a distribution over the images given that it should be labeled as something.  This is often a *much* simpler distribution to train, but with the expense of it being much harder to collect.\n",
    "\n",
    "One of the major directions of future research will be in how to better make use of unlabeled data and unsupervised learning methods.\n",
    "\n",
    "<a name=\"part-one---autoencoders\"></a>\n",
    "# Part One - Autoencoders\n",
    "\n",
    "<a name=\"instructions\"></a>\n",
    "## Instructions\n",
    "\n",
    "Work with a dataset of images and train an autoencoder.  You can work with the same dataset from assignment 1, or try a larger dataset.  But be careful with the image sizes, and make sure to keep it relatively small (e.g. < 200 x 200 px).  \n",
    "\n",
    "Recall from the lecture that autoencoders are great at \"compressing\" information.  The network's construction and cost function are just like what we've done in the last session.  The network is composed of a series of matrix multiplications and nonlinearities.  The only difference is the output of the network has exactly the same shape as what is input.  This allows us to train the network by saying that the output of the network needs to be just like the input to it, so that it tries to \"compress\" all the information in that video.\n",
    "\n",
    "Autoencoders have some great potential for creative applications, as they allow us to compress a dataset of information and even *generate* new data from that encoding.  We'll see exactly how to do this with a basic autoencoder, and then you'll be asked to explore some of the extensions to produce your own encodings.\n",
    "\n",
    "<a name=\"code\"></a>\n",
    "## Code\n",
    "\n",
    "We'll now go through the process of building an autoencoder just like in the lecture.  First, let's load some data.  You can use the first 100 images of the Celeb Net, your own dataset, or anything else approximately under 1,000 images.  Make sure you resize the images so that they are <= 200x200 pixels, otherwise the training will be *very* slow, and the montages we create will be too large.\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# See how this works w/ Celeb Images or try your own dataset instead:\n",
    "imgs = ...\n",
    "\n",
    "# Then convert the list of images to a 4d array (e.g. use np.array to convert a list to a 4d array):\n",
    "Xs = ...\n",
    "\n",
    "print(Xs.shape)\n",
    "assert(Xs.ndim == 4 and Xs.shape[1] <= 250 and Xs.shape[2] <= 250)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll now make use of something I've written to help us store this data.  It provides some interfaces for generating \"batches\" of data, as well as splitting the data into training, validation, and testing sets.  To use it, we pass in the data and optionally its labels.  If we don't have labels, we just pass in the data.  In the second half of this notebook, we'll explore using a dataset's labels as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ds = datasets.Dataset(Xs)\n",
    "# ds = datasets.CIFAR10(flatten=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It allows us to easily find the mean:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "mean_img = ds.mean().astype(np.uint8)\n",
    "plt.imshow(mean_img)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or the deviation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "std_img = ds.std().astype(np.uint8)\n",
    "plt.imshow(std_img)\n",
    "print(std_img.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall we can calculate the mean of the standard deviation across each color channel:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "std_img = np.mean(std_img, axis=2).astype(np.uint8)\n",
    "plt.imshow(std_img)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the input data we gave as input to our `Datasets` object, previously stored in `Xs` is now stored in a variable as part of our `ds` Datasets object, `X`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.imshow(ds.X[0])\n",
    "print(ds.X.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It takes a parameter, `split` at the time of creation, which allows us to create train/valid/test sets.  By default, this is set to `[1.0, 0.0, 0.0]`, which means to take all the data in the train set, and nothing in the validation and testing sets.  We can access \"batch generators\" of each of these sets by saying: `ds.train.next_batch`.  A generator is a really powerful way of handling iteration in Python.  If you are unfamiliar with the idea of generators, I recommend reading up a little bit on it, e.g. here: http://intermediatepythonista.com/python-generators - think of it as a for loop, but as a function.  It returns one iteration of the loop each time you call it.\n",
    "\n",
    "This generator will automatically handle the randomization of the dataset.  Let's try looping over the dataset using the batch generator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for (X, y) in ds.train.next_batch(batch_size=10):\n",
    "    print(X.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This returns `X` and `y` as a tuple.  Since we're not using labels, we'll just ignore this.  The `next_batch` method takes a parameter, `batch_size`, which we'll set appropriately to our batch size.  Notice it runs for exactly 10 iterations to iterate over our 100 examples, then the loop exits.  The order in which it iterates over the 100 examples is randomized each time you iterate.\n",
    "\n",
    "Write two functions to preprocess (normalize) any given image, and to unprocess it, i.e. unnormalize it by removing the normalization.  The `preprocess` function should perform exactly the task you learned to do in assignment 1: subtract the mean, then divide by the standard deviation.  The `deprocess` function should take the preprocessed image and undo the preprocessing steps.  Recall that the `ds` object contains the `mean` and `std` functions for access the mean and standarad deviation.  We'll be using the `preprocess` and `deprocess` functions on the input and outputs of the network.  Note, we could use Tensorflow to do this instead of numpy, but for sake of clarity, I'm keeping this separate from the Tensorflow graph.\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Write a function to preprocess/normalize an image, given its dataset object\n",
    "# (which stores the mean and standard deviation!)\n",
    "def preprocess(img, ds):\n",
    "    norm_img = (img - ...) / ...\n",
    "    return norm_img\n",
    "\n",
    "# Write a function to undo the normalization of an image, given its dataset object\n",
    "# (which stores the mean and standard deviation!)\n",
    "def deprocess(norm_img, ds):\n",
    "    img = norm_img * ... + ...\n",
    "    return img"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're going to now work on creating an autoencoder.  To start, we'll only use linear connections, like in the last assignment.  This means, we need a 2-dimensional input:  Batch Size x Number of Features.  We currently have a 4-dimensional input: Batch Size x Height x Width x Channels.  We'll have to calculate the number of features we have to help construct the Tensorflow Graph for our autoencoder neural network.  Then, when we are ready to train the network, we'll reshape our 4-dimensional dataset into a 2-dimensional one when feeding the input of the network.  Optionally, we could create a `tf.reshape` as the first operation of the network, so that we can still pass in our 4-dimensional array, and the Tensorflow graph would reshape it for us.  We'll try the former method, by reshaping manually, and then you can explore the latter method, of handling 4-dimensional inputs on your own.\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Calculate the number of features in your image.\n",
    "# This is the total number of pixels, or (height x width x channels).\n",
    "n_features = ...\n",
    "print(n_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a list of how many neurons we want in each layer.  This should be for just one half of the network, the encoder only.  It should start large, then get smaller and smaller.  We're also going to try an encode our dataset to an inner layer of just 2 values.  So from our number of features, we'll go all the way down to expressing that image by just 2 values.  Try the values I've put here for the celeb dataset, then explore your own values:\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "encoder_dimensions = [2048, 512, 128, 2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now create a placeholder just like in the last session in the tensorflow graph that will be able to get any number (None) of `n_features` inputs.\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X = tf.placeholder(...\n",
    "                   \n",
    "assert(X.get_shape().as_list() == [None, n_features])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now complete the function `encode` below.  This takes as input our input placeholder, `X`, our list of `dimensions`, and an `activation` function, e.g. `tf.nn.relu` or `tf.nn.tanh`, to apply to each layer's output, and creates a series of fully connected layers.  This works just like in the last session!  We multiply our input, add a bias, then apply a non-linearity.  Instead of having 20 neurons in each layer, we're going to use our `dimensions` list to tell us how many neurons we want in each layer.\n",
    "\n",
    "One important difference is that we're going to also store every weight matrix we create!  This is so that we can use the same weight matrices when we go to build our decoder.  This is a *very* powerful concept that creeps up in a few different neural network architectures called weight sharing.  Weight sharing isn't necessary to do of course, but can speed up training and offer a different set of features depending on your dataset.  Explore trying both.  We'll also see how another form of weight sharing works in convolutional networks.\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def encode(X, dimensions, activation=tf.nn.tanh):\n",
    "    # We're going to keep every matrix we create so let's create a list to hold them all\n",
    "    Ws = []\n",
    "\n",
    "    # We'll create a for loop to create each layer:\n",
    "    for layer_i, n_output in enumerate(dimensions):\n",
    "\n",
    "        # TODO: just like in the last session,\n",
    "        # we'll use a variable scope to help encapsulate our variables\n",
    "        # This will simply prefix all the variables made in this scope\n",
    "        # with the name we give it.  Make sure it is a unique name\n",
    "        # for each layer, e.g., 'encoder/layer1', 'encoder/layer2', or\n",
    "        # 'encoder/1', 'encoder/2',... \n",
    "        with tf.variable_scope(...)\n",
    "\n",
    "            # TODO: Create a weight matrix which will increasingly reduce\n",
    "            # down the amount of information in the input by performing\n",
    "            # a matrix multiplication.  You can use the utils.linear function.\n",
    "            h, W = ...\n",
    "\n",
    "            # Finally we'll store the weight matrix.\n",
    "            # We need to keep track of all\n",
    "            # the weight matrices we've used in our encoder\n",
    "            # so that we can build the decoder using the\n",
    "            # same weight matrices.\n",
    "            Ws.append(W)\n",
    "            \n",
    "            # Replace X with the current layer's output, so we can\n",
    "            # use it in the next layer.\n",
    "            X = h\n",
    "    \n",
    "    z = X\n",
    "    return Ws, z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Then call the function\n",
    "Ws, z = encode(X, encoder_dimensions)\n",
    "\n",
    "# And just some checks to make sure you've done it right.\n",
    "assert(z.get_shape().as_list() == [None, 2])\n",
    "assert(len(Ws) == len(encoder_dimensions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the graph:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "[op.name for op in tf.get_default_graph().get_operations()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we've created a few layers, encoding our input `X` all the way down to 2 values in the tensor `z`.  We do this by multiplying our input `X` by a set of matrices shaped as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "[W_i.get_shape().as_list() for W_i in Ws]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Resulting in a layer which is shaped as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "z.get_shape().as_list()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the Decoder \n",
    "\n",
    "Here is a helpful animation on what the matrix \"transpose\" operation does:\n",
    "![transpose](https://upload.wikimedia.org/wikipedia/commons/e/e4/Matrix_transpose.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basically what is happening is rows becomes columns, and vice-versa.  We're going to use our existing weight matrices but transpose them so that we can go in the opposite direction.  In order to build our decoder, we'll have to do the opposite of what we've just done, multiplying `z` by the transpose of our weight matrices, to get back to a reconstructed version of `X`.  First, we'll reverse the order of our weight matrics, and then append to the list of dimensions the final output layer's shape to match our input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# We'll first reverse the order of our weight matrices\n",
    "decoder_Ws = Ws[::-1]\n",
    "\n",
    "# then reverse the order of our dimensions\n",
    "# appending the last layers number of inputs.\n",
    "decoder_dimensions = encoder_dimensions[::-1][1:] + [n_features]\n",
    "print(decoder_dimensions)\n",
    "\n",
    "assert(decoder_dimensions[-1] == n_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll build the decoder.  I've shown you how to do this.  Read through the code to fully understand what it is doing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def decode(z, dimensions, Ws, activation=tf.nn.tanh):\n",
    "    current_input = z\n",
    "    for layer_i, n_output in enumerate(dimensions):\n",
    "        # we'll use a variable scope again to help encapsulate our variables\n",
    "        # This will simply prefix all the variables made in this scope\n",
    "        # with the name we give it.\n",
    "        with tf.variable_scope(\"decoder/layer/{}\".format(layer_i)):\n",
    "\n",
    "            # Now we'll grab the weight matrix we created before and transpose it\n",
    "            # So a 3072 x 784 matrix would become 784 x 3072\n",
    "            # or a 256 x 64 matrix, would become 64 x 256\n",
    "            W = tf.transpose(Ws[layer_i])\n",
    "\n",
    "            # Now we'll multiply our input by our transposed W matrix\n",
    "            h = tf.matmul(current_input, W)\n",
    "\n",
    "            # And then use a relu activation function on its output\n",
    "            current_input = activation(h)\n",
    "\n",
    "            # We'll also replace n_input with the current n_output, so that on the\n",
    "            # next iteration, our new number inputs will be correct.\n",
    "            n_input = n_output\n",
    "    Y = current_input\n",
    "    return Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Y = decode(z, decoder_dimensions, decoder_Ws)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the new operations we've just added.  They will all be prefixed by \"decoder\" so we can use list comprehension to help us with this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "[op.name for op in tf.get_default_graph().get_operations()\n",
    " if op.name.startswith('decoder')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's take a look at the output of the autoencoder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Y.get_shape().as_list()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great!  So we should have a synthesized version of our input placeholder, `X`, inside of `Y`.  This `Y` is the result of many matrix multiplications, first a series of multiplications in our encoder all the way down to 2 dimensions, and then back to the original dimensions through our decoder.  Let's now create a pixel-to-pixel measure of error.  This should measure the difference in our synthesized output, `Y`, and our input, `X`.  You can use the $l_1$ or $l_2$ norm, just like in assignment 2.  If you don't remember, go back to homework 2 where we calculated the cost function and try the same idea here.\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Calculate some measure of loss, e.g. the pixel to pixel absolute difference or squared difference\n",
    "loss = ...\n",
    "\n",
    "# Now sum over every pixel and then calculate the mean over the batch dimension (just like session 2!)\n",
    "# hint, use tf.reduce_mean and tf.reduce_sum\n",
    "cost = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now for the standard training code.  We'll pass our `cost` to an optimizer, and then use mini batch gradient descent to optimize our network's parameters.  We just have to be careful to make sure we're preprocessing our input and feed it in the right shape, a 2-dimensional matrix of [batch_size, n_features] in dimensions.\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learning_rate = ...\n",
    "optimizer = tf.train.AdamOptimizer(...).minimize(...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is the training code for our autoencoder.  Please go through each line of code to make sure you understand what is happening, and fill in the missing pieces.  This will take awhile.  On my machine, it takes about 15 minutes.  If you're impatient, you can \"Interrupt\" the kernel by going to the Kernel menu above, and continue with the notebook.  Though, the longer you leave this to train, the better the result will be.\n",
    "\n",
    "What I really want you to notice is what the network learns to encode first, based on what it is able to reconstruct.  It won't able to reconstruct everything.  At first, it will just be the mean image.  Then, other major changes in the dataset.  For the first 100 images of celeb net, this seems to be the background: white, blue, black backgrounds.  From this basic interpretation, you can reason that the autoencoder has learned a representation of the backgrounds, and is able to encode that knowledge of the background in its inner most layer of just two values.  It then goes on to represent the major variations in skin tone and hair.  Then perhaps some facial features such as lips.  So the features it is able to encode tend to be the major things at first, then the smaller things.\n",
    "\n",
    "<h3><font color='red'>TODO! COMPLETE THIS SECTION!</font></h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# (TODO) Create a tensorflow session and initialize all of our weights:\n",
    "sess = ...\n",
    "sess.run(tf.global_variables_initializer())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that if you run into \"InternalError\" or \"ResourceExhaustedError\", it is likely that you have run out of memory!  Try a smaller network!  For instance, restart the notebook's kernel, and then go back to defining `encoder_dimensions = [256, 2]` instead.  If you run into memory problems below, you can also try changing the batch_size to 50."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Some parameters for training\n",
    "batch_size = 100\n",
    "n_epochs = 31\n",
    "step = 10\n",
    "\n",
    "# We'll try to reconstruct the same first 100 images and show how\n",
    "# The network does over the course of training.\n",
    "examples = ds.X[:100]\n",
{'fraction_non_alphanumeric': 0.1383945717577775, 'fraction_numerical': 0.0064012290359749075, 'mean_word_length': 2.7744020294757186, 'pattern_counts': {'":': 467, '<': 141, '<?xml version=': 0, '>': 141, 'https://': 12, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7188920', 'n_tokens_mistral': 20015, 'n_tokens_neox': 19304, 'n_words': 7299}