<commit_before>{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1Pi_B2cvdBiW"
      },
      "source": [
        "##### Copyright 2021 The TF-Agents Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "nQnmcm0oI1Q-"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GiI8CZYWcJ5n"
      },
      "source": [
        "# Networks\n",
        "\n",
        "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://www.tensorflow.org/agents/tutorials/8_networks_tutorial\"\u003e\n",
        "    \u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" /\u003e\n",
        "    View on TensorFlow.org\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/agents/blob/master/docs/tutorials/8_networks_tutorial.ipynb\"\u003e\n",
        "    \u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003e\n",
        "    Run in Google Colab\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/agents/blob/master/docs/tutorials/8_networks_tutorial.ipynb\"\u003e\n",
        "    \u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003e\n",
        "    View source on GitHub\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca href=\"https://storage.googleapis.com/tensorflow_docs/agents/docs/tutorials/8_networks_tutorial.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/download_logo_32px.png\" /\u003eDownload notebook\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "\u003c/table\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "31uij8nIo5bG"
      },
      "source": [
        "## Introduction\n",
        "\n",
        "In this colab we will cover how to define custom networks for your agents. The networks help us define the model that is trained by agents. In TF-Agents you will find several different types of networks which are useful across agents:\n",
        "\n",
        "**Main Networks**\n",
        "\n",
        "* **QNetwork**: Used in Qlearning for environments with discrete actions, this network maps an observation to value estimates for each possible action.\n",
        "* **CriticNetworks**: Also referred to as `ValueNetworks` in literature, learns to estimate some version of a Value function mapping some state into an estimate for the expected return of a policy. These networks estimate how good the state the agent is currently in is.\n",
        "* **ActorNetworks**: Learn a mapping from observations to actions. These networks are usually used by our policies to generate actions.\n",
        "* **ActorDistributionNetworks**: Similar to `ActorNetworks` but these generate a distribution which a policy can then sample to generate actions.\n",
        "\n",
        "**Helper Networks**\n",
        "* **EncodingNetwork**: Allows users to easily define a mapping of pre-processing layers to apply to a network's input.\n",
        "* **DynamicUnrollLayer**: Automatically resets the network's state on episode boundaries as it is applied over a time sequence.\n",
        "* **ProjectionNetwork**: Networks like `CategoricalProjectionNetwork` or `NormalProjectionNetwork` take inputs and generate the required parameters to generate Categorical, or Normal distributions.\n",
        "\n",
        "All examples in TF-Agents come with pre-configured networks. However these networks are not setup to handle complex observations.\n",
        "\n",
        "If you have an environment which exposes more than one observation/action and you need to customize your networks then this tutorial is for you!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wmk1GBT9cPqC"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uhGeL1Kpc3Pw"
      },
      "source": [
        "If you haven't installed tf-agents yet, run:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xsLTHlVdiZP3"
      },
      "outputs": [],
      "source": [
        "!pip install tf-agents"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sdvop99JlYSM"
      },
      "outputs": [],
      "source": [
        "from __future__ import absolute_import\n",
        "from __future__ import division\n",
        "from __future__ import print_function\n",
        "\n",
        "import abc\n",
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "\n",
        "from tf_agents.environments import random_py_environment\n",
        "from tf_agents.environments import tf_py_environment\n",
        "from tf_agents.networks import encoding_network\n",
        "from tf_agents.networks import network\n",
        "from tf_agents.networks import utils\n",
        "from tf_agents.specs import array_spec\n",
        "from tf_agents.utils import common as common_utils\n",
        "from tf_agents.utils import nest_utils"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ums84-YP_21F"
      },
      "source": [
        "## Defining Networks\n",
        "\n",
        "### Network API\n",
        "\n",
        "In TF-Agents we subclass from Keras [Networks](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/keras/engine/network.py). With it we can:\n",
        "\n",
        "* Simplify copy operations required when creating target networks.\n",
        "* Perform automatic variable creation when calling `network.variables()`.\n",
        "* Validate inputs based on network input_specs.\n",
        "\n",
        "##EncodingNetwork\n",
        "As mentioned above the `EncodingNetwork` allows us to easily define a mapping of pre-processing layers to apply to a network's input to generate some encoding.\n",
        "\n",
        "The EncodingNetwork is composed of the following mostly optional layers:\n",
        "\n",
        "  * Preprocessing layers\n",
        "  * Preprocessing combiner\n",
        "  * Conv2D \n",
        "  * Flatten\n",
        "  * Dense \n",
        "\n",
        "The special thing about encoding networks is that input preprocessing is applied. Input preprocessing is possible via `preprocessing_layers` and `preprocessing_combiner` layers.  Each of these can be specified as a nested structure. If the `preprocessing_layers` nest is shallower than `input_tensor_spec`, then the layers will get the subnests. For example, if:\n",
        "\n",
        "```\n",
        "input_tensor_spec = ([TensorSpec(3)] * 2, [TensorSpec(3)] * 5)\n",
        "preprocessing_layers = (Layer1(), Layer2())\n",<commit_msg>Fixing broken link in Networks_Tutorial<commit_after>{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1Pi_B2cvdBiW"
      },
      "source": [
        "##### Copyright 2021 The TF-Agents Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "nQnmcm0oI1Q-"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GiI8CZYWcJ5n"
      },
      "source": [
        "# Networks\n",
        "\n",
        "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://www.tensorflow.org/agents/tutorials/8_networks_tutorial\"\u003e\n",
        "    \u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" /\u003e\n",
        "    View on TensorFlow.org\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/agents/blob/master/docs/tutorials/8_networks_tutorial.ipynb\"\u003e\n",
        "    \u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003e\n",
        "    Run in Google Colab\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/agents/blob/master/docs/tutorials/8_networks_tutorial.ipynb\"\u003e\n",
        "    \u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003e\n",
        "    View source on GitHub\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca href=\"https://storage.googleapis.com/tensorflow_docs/agents/docs/tutorials/8_networks_tutorial.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/download_logo_32px.png\" /\u003eDownload notebook\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "\u003c/table\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "31uij8nIo5bG"
      },
      "source": [
        "## Introduction\n",
        "\n",
        "In this colab we will cover how to define custom networks for your agents. The networks help us define the model that is trained by agents. In TF-Agents you will find several different types of networks which are useful across agents:\n",
        "\n",
        "**Main Networks**\n",
        "\n",
        "* **QNetwork**: Used in Qlearning for environments with discrete actions, this network maps an observation to value estimates for each possible action.\n",
        "* **CriticNetworks**: Also referred to as `ValueNetworks` in literature, learns to estimate some version of a Value function mapping some state into an estimate for the expected return of a policy. These networks estimate how good the state the agent is currently in is.\n",
        "* **ActorNetworks**: Learn a mapping from observations to actions. These networks are usually used by our policies to generate actions.\n",
        "* **ActorDistributionNetworks**: Similar to `ActorNetworks` but these generate a distribution which a policy can then sample to generate actions.\n",
        "\n",
        "**Helper Networks**\n",
        "* **EncodingNetwork**: Allows users to easily define a mapping of pre-processing layers to apply to a network's input.\n",
        "* **DynamicUnrollLayer**: Automatically resets the network's state on episode boundaries as it is applied over a time sequence.\n",
        "* **ProjectionNetwork**: Networks like `CategoricalProjectionNetwork` or `NormalProjectionNetwork` take inputs and generate the required parameters to generate Categorical, or Normal distributions.\n",
        "\n",
        "All examples in TF-Agents come with pre-configured networks. However these networks are not setup to handle complex observations.\n",
        "\n",
        "If you have an environment which exposes more than one observation/action and you need to customize your networks then this tutorial is for you!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wmk1GBT9cPqC"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uhGeL1Kpc3Pw"
      },
      "source": [
        "If you haven't installed tf-agents yet, run:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xsLTHlVdiZP3"
      },
      "outputs": [],
      "source": [
        "!pip install tf-agents"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sdvop99JlYSM"
      },
      "outputs": [],
      "source": [
        "from __future__ import absolute_import\n",
        "from __future__ import division\n",
        "from __future__ import print_function\n",
        "\n",
        "import abc\n",
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "\n",
        "from tf_agents.environments import random_py_environment\n",
        "from tf_agents.environments import tf_py_environment\n",
        "from tf_agents.networks import encoding_network\n",
        "from tf_agents.networks import network\n",
        "from tf_agents.networks import utils\n",
        "from tf_agents.specs import array_spec\n",
        "from tf_agents.utils import common as common_utils\n",
        "from tf_agents.utils import nest_utils"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ums84-YP_21F"
      },
      "source": [
        "## Defining Networks\n",
        "\n",
        "### Network API\n",
        "\n",
        "In TF-Agents we subclass from Keras [Networks](https://github.com/tensorflow/agents/blob/master/tf_agents/networks/network.py). With it we can:\n",
        "\n",
        "* Simplify copy operations required when creating target networks.\n",
        "* Perform automatic variable creation when calling `network.variables()`.\n",
        "* Validate inputs based on network input_specs.\n",
        "\n",
        "##EncodingNetwork\n",
        "As mentioned above the `EncodingNetwork` allows us to easily define a mapping of pre-processing layers to apply to a network's input to generate some encoding.\n",
        "\n",
        "The EncodingNetwork is composed of the following mostly optional layers:\n",
        "\n",
        "  * Preprocessing layers\n",
        "  * Preprocessing combiner\n",
        "  * Conv2D \n",
        "  * Flatten\n",
        "  * Dense \n",
        "\n",
        "The special thing about encoding networks is that input preprocessing is applied. Input preprocessing is possible via `preprocessing_layers` and `preprocessing_combiner` layers.  Each of these can be specified as a nested structure. If the `preprocessing_layers` nest is shallower than `input_tensor_spec`, then the layers will get the subnests. For example, if:\n",
        "\n",
        "```\n",
        "input_tensor_spec = ([TensorSpec(3)] * 2, [TensorSpec(3)] * 5)\n",
        "preprocessing_layers = (Layer1(), Layer2())\n",
{'fraction_non_alphanumeric': 0.13636933885334337, 'fraction_numerical': 0.02258185923974407, 'mean_word_length': 2.3543025457605724, 'pattern_counts': {'":': 88, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 20, 'lorem ipsum': 0, 'www.': 12, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7575998', 'n_tokens_mistral': 5169, 'n_tokens_neox': 4874, 'n_words': 1333}