<commit_before>{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "grBmytrShbUE"
      },
      "source": [
        "# High-performance simulations with TFF\n",
        "\n",
        "This tutorial will describe how to setup high-performance simulations with TFF\n",
        "in a variety of common scenarios.\n",
        "\n",
        "NOTE: The mechanisms covered here are not included in the latest release, have\n",
        "not been tested yet, and the API may evolve. In order to follow this tutorial,\n",
        "you will need to build a TFF pip package from scratch from the latest sources, and install it in a Jupyter notebook with a Python 3 runtime. The new executor\n",
        "stack is not compatible with Python 2.\n",
        "\n",
        "TODO(b/134543154): Populate the content, some of the things to cover here:\n",
        "- using GPUs in a single-machine setup,\n",
        "- multi-machine setup on GCP/GKE, with and without TPUs,\n",
        "- interfacing MapReduce-like backends,\n",
        "- current limitations and when/how they will be relaxed."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "yiq_MY4LopET"
      },
      "source": [
        "## Before we begin\n",
        "\n",
        "First, make sure your notebook is connected to a backend that has the relevant\n",
        "components (including gRPC dependencies for multi-machine scenarios) compiled.\n",
        "\n",
        "Now, if you are running this notebook in Jupyter, you may need to take an extra\n",
        "step to work around the\n",
        "[limitations of Jupyter with asyncio](https://github.com/jupyter/notebook/issues/3397#issuecomment-419386811)\n",
        "by installing the [nest_asyncio](https://github.com/erdewit/nest_asyncio)\n",
        "package."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "tnngjncsPq15"
      },
      "outputs": [],
      "source": [
        "!pip install --upgrade nest_asyncio\n",
        "import nest_asyncio\n",
        "nest_asyncio.apply()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "_zFenI3IPpgI"
      },
      "source": [
        "Now, let's start by loading the MNIST example from the TFF website, and\n",
        "declaring the Python function that will run a small experiment loop over\n",
        "a group of 10 clients."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "ke7EyuvG0Zyn"
      },
      "outputs": [],
      "source": [
        "!pip install --upgrade tensorflow_federated\n",
        "!pip install --upgrade tf-nightly"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "2dVPgxN0MdG2"
      },
      "outputs": [],
      "source": [
        "import collections\n",
        "import warnings\n",
        "import time\n",
        "\n",
        "import tensorflow as tf\n",
        "tf.compat.v1.enable_v2_behavior()\n",
        "\n",
        "import tensorflow_federated as tff\n",
        "\n",
        "warnings.simplefilter('ignore')\n",
        "\n",
        "source, _ = tff.simulation.datasets.emnist.load_data()\n",
        "\n",
        "def map_fn(example):\n",<commit_msg>Add automated test for simulations tutorial<commit_after>{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "grBmytrShbUE"
      },
      "source": [
        "# High-performance simulations with TFF\n",
        "\n",
        "This tutorial will describe how to setup high-performance simulations with TFF\n",
        "in a variety of common scenarios.\n",
        "\n",
        "NOTE: The mechanisms covered here are not included in the latest release, have\n",
        "not been tested yet, and the API may evolve. In order to follow this tutorial,\n",
        "you will need to build a TFF pip package from scratch from the latest sources, and install it in a Jupyter notebook with a Python 3 runtime. The new executor\n",
        "stack is not compatible with Python 2.\n",
        "\n",
        "TODO(b/134543154): Populate the content, some of the things to cover here:\n",
        "- using GPUs in a single-machine setup,\n",
        "- multi-machine setup on GCP/GKE, with and without TPUs,\n",
        "- interfacing MapReduce-like backends,\n",
        "- current limitations and when/how they will be relaxed."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "yiq_MY4LopET"
      },
      "source": [
        "## Before we begin\n",
        "\n",
        "First, make sure your notebook is connected to a backend that has the relevant\n",
        "components (including gRPC dependencies for multi-machine scenarios) compiled.\n",
        "\n",
        "Now, if you are running this notebook in Jupyter, you may need to take an extra\n",
        "step to work around the\n",
        "[limitations of Jupyter with asyncio](https://github.com/jupyter/notebook/issues/3397#issuecomment-419386811)\n",
        "by installing the [nest_asyncio](https://github.com/erdewit/nest_asyncio)\n",
        "package."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "tnngjncsPq15"
      },
      "outputs": [],
      "source": [
        "#@test {\"skip\": true}\n",
        "!pip install --upgrade nest_asyncio\n",
        "import nest_asyncio\n",
        "nest_asyncio.apply()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "_zFenI3IPpgI"
      },
      "source": [
        "Now, let's start by loading the MNIST example from the TFF website, and\n",
        "declaring the Python function that will run a small experiment loop over\n",
        "a group of 10 clients."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "ke7EyuvG0Zyn"
      },
      "outputs": [],
      "source": [
        "#@test {\"skip\": true}\n",
        "!pip install --upgrade tensorflow_federated\n",
        "!pip install --upgrade tf-nightly"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "2dVPgxN0MdG2"
      },
      "outputs": [],
      "source": [
        "import collections\n",
        "import warnings\n",
        "import time\n",
        "\n",
        "import tensorflow as tf\n",
        "tf.compat.v1.enable_v2_behavior()\n",
        "\n",
        "import tensorflow_federated as tff\n",
        "\n",
        "warnings.simplefilter('ignore')\n",
        "\n",
        "source, _ = tff.simulation.datasets.emnist.load_data()\n",
        "\n",
        "def map_fn(example):\n",
{'fraction_non_alphanumeric': 0.14626129827444537, 'fraction_numerical': 0.010955902492467817, 'mean_word_length': 1.9832516339869282, 'pattern_counts': {'":': 82, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3288334', 'n_tokens_mistral': 2394, 'n_tokens_neox': 2297, 'n_words': 640}