<commit_before>{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "cmrFlYUtD9Tv"
      },
      "source": [
        "##### Copyright 2019 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "-xwaSGkjD9sL"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "oBTK_JR2EBS9"
      },
      "source": [
        "# DeepDream\n",
        "\n",
        "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://www.tensorflow.org/beta/tutorials/generative/deepdream\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" /\u003eView on TensorFlow.org\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/r2/tutorials/generative/deepdream.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003eRun in Google Colab\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/r2/tutorials/generative/deepdream.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003eView source on GitHub\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/en/r2/tutorials/generative/deepdream.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/download_logo_32px.png\" /\u003eDownload notebook\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "\u003c/table\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "HxBfLCOSEBQk"
      },
      "source": [
        "This tutorial contains a minimal implementation of DeepDream, as described in this [blog post](https://ai.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html) by <NAME>.\n",
        "\n",
        "DeepDream is an experiment that visualizes the patterns learned by a neural network. Similar to when a child watches clouds and tries to interpret random shapes, DeepDream over-interprets and enhances the patterns it sees in an image.\n",
        "\n",
        "It does so by forwarding an image through the network, then calculating the gradient of the image with respect to the activations of a particular layer. The image is then modified to increase these activations, enhancing the patterns seen by the network, and resulting in a dream-like image. This process was dubbed \"Inceptionism\" (a reference to [InceptionNet](https://arxiv.org/pdf/1409.4842.pdf), and the movie Inception).\n",
        "\n",
        "Let's demonstrate how you can make a neural network \"dream\" and enhance the surreal patterns it sees in an image."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "lzlNLHf1c1Zl"
      },
      "outputs": [],
      "source": [
        "from __future__ import absolute_import, division, print_function, unicode_literals\n",
        "\n",
        "try:\n",
        "  # %tensorflow_version only exists in Colab.\n",
        "  %tensorflow_version 2.x\n",
        "except Exception:\n",
        "  pass\n",
        "import tensorflow as tf\n",
        "\n",
        "from IPython.display import clear_output\n",
        "import matplotlib as mpl\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "mpl.rcParams['figure.figsize'] = (8, 8)\n",
        "mpl.rcParams['axes.grid'] = False"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "kE7Kx_R_NQfs"
      },
      "source": [
        "## Choose an image\n",
        "For this tutorial, let's use the image of a [Green Sea Turtle](https://commons.wikimedia.org/wiki/File:Green_Sea_Turtle_grazing_seagrass.jpg)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "FWUosOGeM2zz"
      },
      "outputs": [],
      "source": [
        "url = 'https://storage.googleapis.com/download.tensorflow.org/example_images/Green_Sea_Turtle_grazing_seagrass.jpg'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "ufdACYo-NNmr"
      },
      "outputs": [],
      "source": [
        "# Download an image and read it into a NumPy array.\n",
        "def download(url):\n",
        "  name = url.split('/')[-1]\n",
        "  image_path = tf.keras.utils.get_file(name, origin=url)\n",
        "  img = tf.keras.preprocessing.image.load_img(image_path)\n",
        "  return img\n",
        "\n",
        "# Normalize an image\n",
        "def deprocess(img):\n",
        "  return (img + 1.0)/2.0\n",
        "\n",
        "original_img = download(url)\n",
        "plt.imshow(original_img)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "n92JLO0_S1PO"
      },
      "source": [
        "## Prepare the feature extraction model\n",
        "\n",
        "Let's load a pre-trained image classification model. For this task, a pre-trained [InceptionV3](https://keras.io/applications/#inceptionv3) is used, similar to the network originally used by the authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "YTDZ403gSUIK"
      },
      "outputs": [],
      "source": [
        "base_model = tf.keras.applications.InceptionV3(include_top=False, weights='imagenet')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "MnOHAiOdUrh8"
      },
      "source": [
        "The idea in DeepDream is to choose a layer (or layers) and maximize the \"loss\" in a way that the image increasingly \"excites\" the layers. The complexity of the features incorporated depends on layers chosen by you, i.e, lower layers produce strokes or simple patterns, while deeper layers give sophisticated features in images, or even whole objects."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "MeztYKpJUMxd"
      },
      "outputs": [],
      "source": [
        "# Maximize the activations of these arbitarily chosen layers\n",
        "names = ['mixed2', 'mixed3', 'mixed5']\n",
        "layers = [base_model.get_layer(name).output for name in names]\n",
        "\n",
        "# Create our feature extraction model\n",
        "dream_model = tf.keras.Model(inputs=base_model.input, outputs=layers)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "g3pin_-rMiq_"
      },
      "source": [
        "## Calculate loss\n",
        "\n",
        "The loss is the sum of the activations in the chosen layers. The loss is normalizaed at each layer so the contribution from larger layers does not outweigh smaller layers. Normally, loss is a quantity you wish to minimize via gradient descent. In DeepDream, you will maximize this loss via gradient ascent."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "p8NeOlixXivC"
      },
      "outputs": [],
      "source": [
        "def calc_loss(image):\n",
        "  image = image[None, ...]\n",
        "  layer_activations = dream_model(image)\n",
        "\n",
        "  losses = []\n",
        "  for act in layer_activations:\n",
        "    loss = tf.math.reduce_mean(act)\n",
        "    losses.append(loss)\n",
        "  total_loss = tf.add_n(losses)\n",
        "  return total_loss"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "k4TCNsAUO9kI"
      },
      "source": [
        "## Gradient ascent\n",
        "\n",
        "Once you have calculated the loss for the chosen layers, all that is left is to calculate the gradients with respect to the image, and add them to the original image. \n",
        "\n",
        "Adding the gradients to the image enhances the patterns seen by the network. At each step, you will have created an image that increasingly excites the activations of certain layers in the network."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "kCIQ_Wi7z4tA"
      },
      "outputs": [],
      "source": [
        "img = tf.keras.preprocessing.image.img_to_array(original_img)\n",
        "img = tf.Variable(tf.keras.applications.inception_v3.preprocess_input(img))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "8YevwXuUN7Ia"
      },
      "outputs": [],
      "source": [
        "optimizer = tf.keras.optimizers.Adam(learning_rate=0.005)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "BaQqrY9IzQqg"
      },
      "outputs": [],
      "source": [
        "@tf.function\n",
        "def deepdream():\n",
        "  with tf.GradientTape() as tape:\n",
        "    loss = -calc_loss(img)\n",
        "  gradient = tape.gradient(loss, img)\n",
        "\n",
        "  optimizer.apply_gradients([(gradient, img)])\n",
        "  img.assign(tf.clip_by_value(img, -1, 1))\n",
        "  return loss"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "UUtC3T2zOBPY"
      },
      "source": [
        "## Result\n",
        "Brace yourself, things are going to get dreamy!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "zw--hysApaHi"
      },
      "outputs": [],
      "source": [
        "STEPS = 500"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "1YRGln2Ca7gs"
      },
      "outputs": [],
      "source": [
        "for step in range(STEPS):\n",
        "  loss = deepdream()\n",
        "  if step % 50 == 0:\n",
        "    clear_output(wait=True)\n",
        "    plt.imshow(deprocess(img))\n",
        "    plt.show()\n",
        "    print ('Step %d, loss %f' % (step, -1.0*loss))\n",
        "clear_output()\n",
        "plt.imshow(deprocess(img))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "D7GYrE7gP-FL"
      },
      "source": [
        "## Next steps\n",
        "\n",
        "Now that you have seen how to make a network \"dream\", go ahead and experiment with a different combination of layers, a different [pre-trained model](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/applications) or your own neural network. You may also modify this code to maximize the activations of certain filters within a layer, as opposed to the layer as a whole.\n",
        "\n",
        "Finally, you may wish to see this complete [implementation](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples/tutorials/deepdream) in an older version of TensorFlow which includes additional tricks."
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "deepdream.ipynb",
      "private_outputs": true,
      "provenance": [],
      "toc_visible": true,
      "version": "0.3.2"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}<commit_msg>Updates deepdream tutorial with octaves and improves "dream" quality.<commit_after>{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "1SgrstLXNbG_"
      },
      "source": [
        "##### Copyright 2019 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "k7gifg92NbG9"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "dCMqzy7BNbG9"
      },
      "source": [
        "# DeepDream"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "2yqCPS8SNbG8"
      },
      "source": [
        "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://www.tensorflow.org/beta/tutorials/generative/deepdream\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" /\u003eView on TensorFlow.org\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/r2/tutorials/generative/deepdream.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003eRun in Google Colab\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/r2/tutorials/generative/deepdream.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003eView source on GitHub\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/en/r2/tutorials/generative/deepdream.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/download_logo_32px.png\" /\u003eDownload notebook\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "\u003c/table\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "XPDKhwPcNbG7"
      },
      "source": [
        "This tutorial contains a minimal implementation of DeepDream, as described in this [blog post](https://ai.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html) by <NAME>.\n",
        "\n",
        "DeepDream is an experiment that visualizes the patterns learned by a neural network. Similar to when a child watches clouds and tries to interpret random shapes, DeepDream over-interprets and enhances the patterns it sees in an image.\n",
        "\n",
        "It does so by forwarding an image through the network, then calculating the gradient of the image with respect to the activations of a particular layer. The image is then modified to increase these activations, enhancing the patterns seen by the network, and resulting in a dream-like image. This process was dubbed \"Inceptionism\" (a reference to [InceptionNet](https://arxiv.org/pdf/1409.4842.pdf), and the [movie](https://en.wikipedia.org/wiki/Inception) Inception.\n",
        "\n",
        "Let's demonstrate how you can make a neural network \"dream\" and enhance the surreal patterns it sees in an image."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "g_Qp173_NbG5"
      },
      "outputs": [],
      "source": [
        "from __future__ import absolute_import, division, print_function, unicode_literals\n",
        "\n",
        "try:\n",
        "  # %tensorflow_version only exists in Colab.\n",
        "  %tensorflow_version 2.x\n",
        "except Exception:\n",
        "  pass\n",
        "import tensorflow as tf\n",
        "\n",
        "import matplotlib as mpl\n",
        "\n",
        "from IPython.display import clear_output\n",
        "from matplotlib import pyplot as plt\n",
        "from tensorflow.keras.preprocessing import image"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "wgeIJg82NbG4"
      },
      "source": [
        "## Choose an image to dream-ify"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "yt6zam_9NbG4"
      },
      "source": [
        "For this tutorial, let's use an image of <NAME>'s [Composition 7](https://commons.wikimedia.org/wiki/File:Vassily_Kandinsky,_1913_-_Composition_7.jpg)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "0lclzk9sNbG2"
      },
      "outputs": [],
      "source": [
        "url = 'https://storage.googleapis.com/download.tensorflow.org/example_images/Vassily_Kandinsky%2C_1913_-_Composition_7.jpg'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Y5BPgc8NNbG0"
      },
      "outputs": [],
      "source": [
        "# Download an image and read it into a NumPy array.\n",
        "def download(url, target_size=None):\n",
        "  name = url.split('/')[-1]\n",
        "  image_path = tf.keras.utils.get_file(name, origin=url)\n",
        "  img = tf.keras.preprocessing.image.load_img(image_path, target_size=target_size)\n",
        "  return img\n",
        "\n",
        "# Normalize an image\n",
        "def deprocess(img):\n",
        "  return (img + 1.0)/2.0\n",
        "\n",
        "# Display an image\n",
        "def show(img):\n",
        "  plt.figure(figsize=(12,12))\n",
        "  plt.grid(False)\n",
        "  plt.axis('off')\n",
        "  plt.imshow(img)\n",
        "  plt.show()\n",
        "\n",
        "# Downsizing the image makes it easier to work with.\n",
        "original_img = download(url, target_size=[225, 375])\n",
        "show(original_img)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "F4RBFfIWNbG0"
      },
      "source": [
        "## Prepare the feature extraction model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "cruNQmMDNbGz"
      },
      "source": [
        "Download and prepare a pre-trained image classification model. You will use [InceptionV3](https://keras.io/applications/#inceptionv3) which is similar to the model originally used in DeepDream. Note that any [pre-trained model](https://keras.io/applications/#models-for-image-classification-with-weights-trained-on-imagenet) will work, although you will have to adjust the layer names below if you change this."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "GlLi48GKNbGy"
      },
      "outputs": [],
      "source": [
        "base_model = tf.keras.applications.InceptionV3(include_top=False, weights='imagenet')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Bujb0jPNNbGx"
      },
      "source": [
        "The idea in DeepDream is to choose a layer (or layers) and maximize the \"loss\" in a way that the image increasingly \"excites\" the layers. The complexity of the features incorporated depends on layers chosen by you, i.e, lower layers produce strokes or simple patterns, while deeper layers give sophisticated features in images, or even whole objects."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "qOVmDO4LNbGv"
      },
      "source": [
        "The InceptionV3 architecture is quite large (for a graph of the model architecture see TensorFlow's [research repo](https://github.com/tensorflow/models/tree/master/research/inception)). For DeepDream, the layers of  interest are those where the convolutions are concatenated. There are 11 of these layers in InceptionV3, named 'mixed0' though 'mixed10'. Using different layers will result in different dream-like images. Deeper layers respond to higher-level features (such as eyes and faces), while earlier layers respond to simpler features (such as edges, shapes, and textures). Feel free to experiment with the layers selected below, but keep in mind that deeper layers (those with a higher index) will take longer to train on since the gradient computation is deeper."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "08KB502ONbGt"
      },
      "outputs": [],
      "source": [
        "# Maximize the activations of these layers\n",
        "names = ['mixed3', 'mixed5']\n",
        "layers = [base_model.get_layer(name).output for name in names]\n",
        "\n",
        "# Create the feature extraction model\n",
        "dream_model = tf.keras.Model(inputs=base_model.input, outputs=layers)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "sb7u31B4NbGt"
      },
      "source": [
        "## Calculate loss\n",
        "\n",
        "The loss is the sum of the activations in the chosen layers. The loss is normalizaed at each layer so the contribution from larger layers does not outweigh smaller layers. Normally, loss is a quantity you wish to minimize via gradient descent. In DeepDream, you will maximize this loss via gradient ascent."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "8MhfSweXXiuq"
      },
      "outputs": [],
      "source": [
        "def calc_loss(img, model):\n",
        "  \n",
        "  # Pass forward the image through the model to retrieve the activations.\n",
        "  # Converts the image into a batch of size 1.\n",
        "  img_batch = tf.expand_dims(img, axis=0)\n",
        "  layer_activations = model(img_batch)\n",
        "\n",
        "  losses = []\n",
        "  for act in layer_activations:\n",
        "    loss = tf.math.reduce_mean(act)\n",
        "    losses.append(loss)\n",
        "\n",
        "  return  tf.reduce_sum(losses)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "k4TCNsAUO9kI"
      },
      "source": [
        "## Gradient ascent\n",
        "\n",
        "Once you have calculated the loss for the chosen layers, all that is left is to calculate the gradients with respect to the image, and add them to the original image. \n",
        "\n",
        "Adding the gradients to the image enhances the patterns seen by the network. At each step, you will have created an image that increasingly excites the activations of certain layers in the network."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "qRScWg_VNqvj"
      },
      "outputs": [],
      "source": [
        "@tf.function\n",
        "def deepdream(model, img_var, step_size):\n",
        "    with tf.GradientTape() as tape:\n",
        "      loss = calc_loss(img_var, model)\n",
        "\n",
        "    # Calculate the gradient of the loss with respect to the pixels of the input image.\n",
        "    gradients = tape.gradient(loss, img_var)\n",
        "\n",
        "    # Normalize the gradients.\n",
        "    gradients /= tf.math.reduce_std(gradients) + 1e-8 \n",
        "    \n",
        "    # In gradient ascent, the \"loss\" is maximized so that the input image increasingly \"excites\" the layers.\n",
        "    # You can update the image by directly adding the gradients (because they're the same shape!)\n",
        "    img_var.assign_add(gradients*step_size)\n",
        "    img_var.assign(tf.clip_by_value(img_var, -1, 1))\n",
        "\n",
        "    return loss"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "9vHEcy7dTysi"
      },
      "outputs": [],
      "source": [
        "def run_deep_dream_simple(model, img, steps=100, step_size=0.01):\n",
        "  # Convert the image into a variable for training\n",
        "  img_arr = tf.keras.preprocessing.image.img_to_array(img)\n",
        "  img_var = tf.Variable(tf.keras.applications.inception_v3.preprocess_input(img_arr))\n",
        "\n",
        "  for step in range(steps):\n",
        "    loss = deepdream(model, img_var, step_size)\n",
        "    \n",
        "    if step % 100 == 0:\n",
        "      clear_output(wait=True)\n",
        "      print (\"Step {}, loss {}\".format(step, loss))\n",
        "      show(deprocess(img_var))\n",
        "\n",
        "  return deprocess(img_var)\n",
        "\n",
        "deep_img = run_deep_dream_simple(model=dream_model, img=original_img, steps=800, step_size=0.001)\n",
        "\n",
        "clear_output()\n",
        "show(deep_img)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "2PbfXEVFNbGp"
      },
      "source": [
        "# Taking it up an octave\n",
        "\n",
        "Pretty good! But the patterns appear like they're all happening at the same granularity. You can improve upon this by appying gradient descent at different scales. This will allow patterns generated at smaller scales to be incorporated into patterns at higher scales and filled in with additional detail.\n",
        "\n",
        "To do this you can perform the previous gradient ascent approach, then increase the size of the image (which is reffered to as an octave), and repeat this process for multiple octaves. One thing to consider is that as the image increases in size, so will the time and memory necessary to perform the gradient calculation. To avoid this issue you can split the image into tiles and compute the gradient for each tile, as is shown below. Applying random shifts to the image before each tiled computation prevents tile seams from appearing."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "x__TZ0uqNbGm"
      },
      "outputs": [],
      "source": [
        "def get_tiled_gradients(model, img, tile_size=512):\n",
        "  # Randomly shift the image to avoid tiled boundaries.\n",
        "  shift_down, shift_right = tf.random.uniform([2], 0, tile_size, dtype=tf.int32)\n",
        "  img = tf.roll(tf.roll(img, shift_right, axis=1), shift_down, axis=0)\n",
        "\n",
        "  # Initialize the image gradients to zero.\n",
        "  gradients = tf.Variable(tf.zeros_like(img))\n",
        "\n",
        "  for x in tf.range(0, img.shape[0], tile_size):\n",
        "    for y in tf.range(0, img.shape[1], tile_size):\n",
        "      # Extract a tile out of the image.\n",
        "      img_tile = tf.Variable(img[x:x+tile_size, y:y+tile_size])\n",
        "\n",
        "      # Calculate the gradients for this tile.\n",
        "      with tf.GradientTape() as tape:\n",
        "        loss = calc_loss(img_tile, model)\n",
        "\n",
        "      # Store the gradients for this tile.\n",
        "      gradients = gradients[x:x+tile_size, y:y+tile_size].assign(tape.gradient(loss, img_tile))\n",
        "\n",
        "  # Undo the random shift applied to the image and its gradients.\n",
        "  gradients = tf.roll(tf.roll(gradients, -shift_right, axis=1), -shift_down, axis=0)\n",
        "\n",
        "  # Normalize the gradients.\n",
        "  gradients /= tf.math.reduce_std(gradients) + 1e-8 \n",
        "\n",
        "  return gradients "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "gA-15DM4NbGk"
      },
      "outputs": [],
      "source": [
        "def run_deep_dream_with_octaves(model, img, steps_per_octave=50, step_size=1.0, \n",
        "                                num_octaves=3, octave_scale=1.3):\n",
        "  # Convert the image into a variable for training\n",
        "  img_arr = tf.keras.preprocessing.image.img_to_array(img)\n",
        "  img_var = tf.Variable(tf.keras.applications.inception_v3.preprocess_input(img_arr))\n",
        "\n",
        "  for octave in range(num_octaves):\n",
        "    # Scale the image based on the octave\n",
        "    if octave\u003e0:\n",
        "      new_size = tf.cast(tf.convert_to_tensor(img_var.shape[:2]), tf.float32)*octave_scale\n",
        "      img_var = tf.Variable(tf.image.resize(img_var, tf.cast(new_size, tf.int32)))\n",
        "\n",
        "    for step in range(steps_per_octave):\n",
        "      gradients = get_tiled_gradients(model, img_var)\n",
        "      img_var.assign_add(gradients*step_size)\n",
        "      img_var.assign(tf.clip_by_value(img_var, -1, 1))\n",
        "\n",
        "      if step % 10 == 0:\n",
        "        clear_output(wait=True)\n",
        "        print (\"Octave {}, Step {}\".format(octave, step))\n",
        "        show(deprocess(img_var))\n",
        "\n",
        "  return deprocess(img_var)\n",
        "\n",
        "deep_img = run_deep_dream_with_octaves(model=dream_model, img=original_img, step_size=0.01)\n",
        "\n",
        "clear_output()\n",
        "show(deep_img)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "0Og0-qLwNbGg"
      },
      "source": [
        "Much better! Play around with the number of octaves, octave scale, and activated layers to change how your DeepDream-ed image looks.\n",
        "\n",
        "Readers might also be interested in [TensorFlow Lucid](https://github.com/tensorflow/lucid) which expands on ideas introduced in this tutorial to visualize and interpret neural networks."
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "last_runtime": {
        "build_target": "//research/colab/notebook:notebook_backend",
        "kind": "private"
      },
      "name": "deepdream.ipynb",
      "private_outputs": true,
      "provenance": [],
      "toc_visible": true,
      "version": "0.3.2"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
{'fraction_non_alphanumeric': 0.1523596050160538, 'fraction_numerical': 0.01787120615496456, 'mean_word_length': 2.1593301435406698, 'pattern_counts': {'":': 339, '<': 6, '<?xml version=': 0, '>': 6, 'https://': 34, 'lorem ipsum': 0, 'www.': 13, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 15, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '521458', 'n_tokens_mistral': 11347, 'n_tokens_neox': 10798, 'n_words': 2588}