<commit_before>{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "LatentConstraints_magenta.ipynb",
      "version": "0.3.2",
      "views": {},
      "default_view": {},
      "provenance": [
        {
          "file_id": "1GPrBZWol4z_5R-LHmOjX3Y8YBdQ-92Rr",
          "timestamp": 1510266173168
        },
        {
          "file_id": "1-RpoOTJxM0AzFskOwW7qXB58sBe4HQJC",
          "timestamp": 1510185306485
        },
        {
          "file_id": "0BxV6RES_0xkhSE5nUTRQbldqTmc",
          "timestamp": 1506463290551
        }
      ],
      "collapsed_sections": [],
      "toc_visible": true
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "oKI30KBUEN6l",
        "colab_type": "text"
      },
      "source": [
        "Copyright 2017 Google LLC.\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "you may not use this file except in compliance with the License.\n",
        "You may obtain a copy of the License at\n",
        "\n",
        "https://www.apache.org/licenses/LICENSE-2.0\n",
        "\n",
        "Unless required by applicable law or agreed to in writing, software\n",
        "distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "See the License for the specific language governing permissions and\n",
        "limitations under the License."
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {
        "id": "8w2CSFQvkALu",
        "colab_type": "text",
        "slideshow": {
          "slide_type": "slide"
        }
      },
      "source": [
        "A version of this notebook with accompanying cloud instance can be found at [https://colab.research.google.com/notebook#fileId=/v2/external/notebooks/latent_constraints/latentconstraints.ipynb](https://colab.research.google.com/notebook#fileId=/v2/external/notebooks/latent_constraints/latentconstraints.ipynb&scrollTo=oKI30KBUEN6l)\n",
        "\n",
        "# Latent Constraints: Conditional Generation from Unconditional Generative Models\n",
        "### ___<NAME>, <NAME>, <NAME>___ [arXiv link](http://arxiv.org/abs/1711.05772)\n",
        "\n",
        "___\n",
        "__Abstract:__ Deep generative neural networks have proven effective at both conditional and unconditional modeling of complex data distributions.\n",
        "Conditional generation enables interactive control, but creating new controls often requires expensive retraining. \n",
        "In this paper, we develop a method to condition generation without retraining the model. \n",
        "By post-hoc learning ___latent constraints___, value functions that identify regions in latent space that generate outputs with desired attributes, we can conditionally sample from these regions with gradient-based optimization or amortized actor functions.\n",
        "Combining attribute constraints with a universal ``realism`` constraint, which enforces similarity to the data distribution, we generate realistic conditional images from an unconditional variational autoencoder.\n",
        "Further, using gradient-based optimization, we demonstrate identity-preserving transformations that make\n",
        "the minimal adjustment in latent space to modify the attributes of an image.\n",
        "Finally, with discrete sequences of musical notes, we demonstrate zero-shot conditional generation, learning latent constraints in the absence of labeled data or a differentiable reward function. \n",
        "\n",
        "___\n",
        "\n",
        "![](http://download.magenta.tensorflow.org/models/latent_constraints/ipynb_figs/diagram.png =400x400 \"Latent Space Diagram\")\n",
        "![](http://download.magenta.tensorflow.org/models/latent_constraints/ipynb_figs/cgen.png =540x500 \"Conditional Generation with and without a distance penalty\")\n",
        "\n",
        "___\n",
        "\n",
        "This notebook contains code for running experiments related to the paper. First, we load pretrained checkpoints:\n",
        "\n",
        "  * VAE models trained on CelebA with pixelwise gaussian data liklihoods of $\\mathcal{N}(\\mu(z), \\sigma_x=0.1)$ and $\\mathcal{N}(\\mu(z), \\sigma_x=1)$. \n",
        "  * We also provide embeddings of the training and eval set from the VAE models.\n",
        "  * A generator ($G$) and discriminator ($D$) from a conditional-GAN, trained to shift samples from the prior to new points in latent space that satisfy the realism constraint ($r$) and attribute constraints ($r_{attr}$). \n",
        "  * We have versions trained with no distance penalty, and also with a penalty of 1e-1.\n",
        "  * A seperately trained attribute classifier in both z-space ($D_{attr}$) and pixel space ($Classifier$).\n",
        "\n",
        "We then proceed to:\n",
        " * Demonstrate that VAE reconstructions sharpen as $\\sigma_x$ lowers, at the expense of sample quality, which is compensated with latent constraints.\n",
        " * Plot conditional generation using CGANs ($D$, $G$) both with and without distance penalty.\n",
        " * Perform identity preserving transformations doing SGD in z-space wrt $D_{attr}$\n",
        " * Evaluate the accuracy of generating images with conditional attributes.\n",
        " \n",
        "Training loops are also provided for demonstration purposes at the end of the notebook.\n",
        "\n",
        "___\n",
        "\n",
        "This colab notebook is self-contained and should run natively on google cloud. The code and checkpoints can be downloaded separately and run locally, which is recommended if you want to train your own model. Pretrained model checkpoints are available at [download.magenta.tensorflow.org/models/latent_constraints/latent_constraints.tar](http://download.magenta.tensorflow.org/models/latent_constraints/latent_constraints.tar). \n",
        "\n",
        "Download them and extract files to `/tmp/` where this notebook assumes that the files exist.\n",
        "\n",
        "__Tips:__ Don't forget you can navigate with the _Table of Contents_ in the left hand sidebar, and collapse all sections with (`Ctrl + Shift + ]`)\n",
        "\n",
        "\n"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {
        "id": "7PMgN4pHd9XD",
        "colab_type": "code",
        "slideshow": {
          "slide_type": "-"
        },
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "# This notebook requires DeepMind's sonnet library, which itself\n",
        "# requires the nightly build of TensorFlow. The command below \n",
        "# installs both.\n",
        "!pip install -q -U dm-sonnet tf-nightly\n",
        "\n",
        "import os\n",
        "import PIL\n",
        "\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import pandas as pd\n",
        "import sklearn.metrics\n",
        "import sonnet as snt\n",
        "import tensorflow as tf\n",
        "\n",
        "ds = tf.contrib.distributions\n",
        "\n",
        "%matplotlib inline"<commit_msg>edit links to colab and images, also edit lib versions (#97)<commit_after>{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "LatentConstraints_magenta.ipynb",
      "version": "0.3.2",
      "views": {},
      "default_view": {},
      "provenance": [
        {
          "file_id": "1GPrBZWol4z_5R-LHmOjX3Y8YBdQ-92Rr",
          "timestamp": 1510266173168
        },
        {
          "file_id": "1-RpoOTJxM0AzFskOwW7qXB58sBe4HQJC",
          "timestamp": 1510185306485
        },
        {
          "file_id": "0BxV6RES_0xkhSE5nUTRQbldqTmc",
          "timestamp": 1506463290551
        }
      ],
      "collapsed_sections": [],
      "toc_visible": true
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "oKI30KBUEN6l",
        "colab_type": "text"
      },
      "source": [
        "Copyright 2017 Google LLC.\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "you may not use this file except in compliance with the License.\n",
        "You may obtain a copy of the License at\n",
        "\n",
        "https://www.apache.org/licenses/LICENSE-2.0\n",
        "\n",
        "Unless required by applicable law or agreed to in writing, software\n",
        "distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "See the License for the specific language governing permissions and\n",
        "limitations under the License."
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {
        "id": "8w2CSFQvkALu",
        "colab_type": "text",
        "slideshow": {
          "slide_type": "slide"
        }
      },
      "source": [
        "A version of this notebook with accompanying cloud instance can be found at [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/notebooks/latent_constraints/latentconstraints.ipynb)\n",
        "\n",
        "# Latent Constraints: Conditional Generation from Unconditional Generative Models\n",
        "### ___<NAME>, <NAME>, <NAME>___ [arXiv link](http://arxiv.org/abs/1711.05772)\n",
        "\n",
        "___\n",
        "__Abstract:__ Deep generative neural networks have proven effective at both conditional and unconditional modeling of complex data distributions.\n",
        "Conditional generation enables interactive control, but creating new controls often requires expensive retraining. \n",
        "In this paper, we develop a method to condition generation without retraining the model. \n",
        "By post-hoc learning ___latent constraints___, value functions that identify regions in latent space that generate outputs with desired attributes, we can conditionally sample from these regions with gradient-based optimization or amortized actor functions.\n",
        "Combining attribute constraints with a universal ``realism`` constraint, which enforces similarity to the data distribution, we generate realistic conditional images from an unconditional variational autoencoder.\n",
        "Further, using gradient-based optimization, we demonstrate identity-preserving transformations that make\n",
        "the minimal adjustment in latent space to modify the attributes of an image.\n",
        "Finally, with discrete sequences of musical notes, we demonstrate zero-shot conditional generation, learning latent constraints in the absence of labeled data or a differentiable reward function. \n",
        "\n",
        "___\n",
        "\n",
        "<img src=\"http://download.magenta.tensorflow.org/models/latent_constraints/ipynb_figs/diagram.png\" alt=\"Latent Space Diagram\" height=400>\n",
        "<img src=\"http://download.magenta.tensorflow.org/models/latent_constraints/ipynb_figs/cgen.png\" alt=\"Conditional Generation with and without a distance penalty\" wight=540 height=500>\n",
        "\n",
        "___\n",
        "\n",
        "This notebook contains code for running experiments related to the paper. First, we load pretrained checkpoints:\n",
        "\n",
        "  * VAE models trained on CelebA with pixelwise gaussian data liklihoods of $\\mathcal{N}(\\mu(z), \\sigma_x=0.1)$ and $\\mathcal{N}(\\mu(z), \\sigma_x=1)$. \n",
        "  * We also provide embeddings of the training and eval set from the VAE models.\n",
        "  * A generator ($G$) and discriminator ($D$) from a conditional-GAN, trained to shift samples from the prior to new points in latent space that satisfy the realism constraint ($r$) and attribute constraints ($r_{attr}$). \n",
        "  * We have versions trained with no distance penalty, and also with a penalty of 1e-1.\n",
        "  * A seperately trained attribute classifier in both z-space ($D_{attr}$) and pixel space ($Classifier$).\n",
        "\n",
        "We then proceed to:\n",
        " * Demonstrate that VAE reconstructions sharpen as $\\sigma_x$ lowers, at the expense of sample quality, which is compensated with latent constraints.\n",
        " * Plot conditional generation using CGANs ($D$, $G$) both with and without distance penalty.\n",
        " * Perform identity preserving transformations doing SGD in z-space wrt $D_{attr}$\n",
        " * Evaluate the accuracy of generating images with conditional attributes.\n",
        " \n",
        "Training loops are also provided for demonstration purposes at the end of the notebook.\n",
        "\n",
        "___\n",
        "\n",
        "This colab notebook is self-contained and should run natively on google cloud. The code and checkpoints can be downloaded separately and run locally, which is recommended if you want to train your own model. Pretrained model checkpoints are available at [download.magenta.tensorflow.org/models/latent_constraints/latent_constraints.tar](http://download.magenta.tensorflow.org/models/latent_constraints/latent_constraints.tar). \n",
        "\n",
        "Download them and extract files to `/tmp/` where this notebook assumes that the files exist.\n",
        "\n",
        "__Tips:__ Don't forget you can navigate with the _Table of Contents_ in the left hand sidebar, and collapse all sections with (`Ctrl + Shift + ]`)\n",
        "\n",
        "\n"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {
        "id": "7PMgN4pHd9XD",
        "colab_type": "code",
        "slideshow": {
          "slide_type": "-"
        },
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "source": [
        "# This notebook requires DeepMind's sonnet library 1.x, which itself\n",
        "# requires TensorFlow 1.x. The command below installs both.\n",
        "!pip install -q -U \"dm-sonnet<2\" \"tensorflow<2\"\n",
        "\n",
        "import os\n",
        "import PIL\n",
        "\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import pandas as pd\n",
        "import sklearn.metrics\n",
        "import sonnet as snt\n",
        "import tensorflow as tf\n",
        "\n",
        "ds = tf.contrib.distributions\n",
        "\n",
        "%matplotlib inline"
{'fraction_non_alphanumeric': 0.11939477303988996, 'fraction_numerical': 0.01485557083906465, 'mean_word_length': 2.5604799216454457, 'pattern_counts': {'":': 80, '<': 13, '<?xml version=': 0, '>': 11, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6258985', 'n_tokens_mistral': 4353, 'n_tokens_neox': 4110, 'n_words': 1377}