<commit_before>{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "uAttKaKmT435"
      },
      "source": [
        "\u003cdiv class=\"devsite-table-wrapper\"\u003e\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "\u003ctd\u003e\u003ca target=\"_blank\" href=\"https://www.tensorflow.org/tfx/tutorials/transform/census\"\u003e\n",
        "\u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" /\u003eView on TensorFlow.org\u003c/a\u003e\u003c/td\u003e\n",
        "\u003ctd\u003e\u003ca target=\"_blank\" href=\"https://colab.sandbox.google.com/github/tensorflow/tfx/blob/master/docs/tutorials/transform/census.ipynb\"\u003e\n",
        "\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\"\u003eRun in Google Colab\u003c/a\u003e\u003c/td\u003e\n",
        "\u003ctd\u003e\u003ca target=\"_blank\" href=\"https://github.com/tensorflow/tfx/blob/master/docs/tutorials/transform/census.ipynb\"\u003e\n",
        "\u003cimg width=32px src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\"\u003eView source on GitHub\u003c/a\u003e\u003c/td\u003e\n",
        "\u003c/table\u003e\u003c/div\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "tghWegsjhpkt"
      },
      "source": [
        "##### Copyright &copy; 2020 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "rSGJWC5biBiG"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "mPt5BHTwy_0F"
      },
      "source": [
        "# Preprocessing data with TensorFlow Transform\n",
        "***The Feature Engineering Component of TensorFlow Extended (TFX)***\n",
        "\n",
        "This example colab notebook provides a somewhat more advanced example of how \u003ca target='_blank' href='https://www.tensorflow.org/tfx/transform/'\u003eTensorFlow Transform\u003c/a\u003e (`tf.Transform`) can be used to preprocess data using exactly the same code for both training a model and serving inferences in production.\n",
        "\n",
        "TensorFlow Transform is a library for preprocessing input data for TensorFlow, including creating features that require a full pass over the training dataset.  For example, using TensorFlow Transform you could:\n",
        "\n",
        "* Normalize an input value by using the mean and standard deviation\n",
        "* Convert strings to integers by generating a vocabulary over all of the input values\n",
        "* Convert floats to integers by assigning them to buckets, based on the observed data distribution\n",
        "\n",
        "TensorFlow has built-in support for manipulations on a single example or a batch of examples. `tf.Transform` extends these capabilities to support full passes over the entire training dataset.\n",
        "\n",
        "The output of `tf.Transform` is exported as a TensorFlow graph which you can use for both training and serving. Using the same graph for both training and serving can prevent skew, since the same transformations are applied in both stages.\n",
        "\n",
        "Key Point: In order to understand `tf.Transform` and how it works with Apache Beam, you'll need to know a little bit about Apache Beam itself.  The \u003ca target='_blank' href='https://beam.apache.org/documentation/programming-guide/'\u003eBeam Programming Guide\u003c/a\u003e is a great place to start."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "_tQUubddMvnP"
      },
      "source": [
        "##What we're doing in this example\n",
        "\n",
        "In this example we'll be processing a \u003ca target='_blank' href='https://archive.ics.uci.edu/ml/machine-learning-databases/adult'\u003ewidely used dataset containing census data\u003c/a\u003e, and training a model to do classification.  Along the way we'll be transforming the data using `tf.Transform`.\n",
        "\n",
        "Key Point: As a modeler and developer, think about how this data is used and the potential benefits and harm a model's predictions can cause. A model like this could reinforce societal biases and disparities. Is a feature relevant to the problem you want to solve or will it introduce bias? For more information, read about \u003ca target='_blank' href='https://developers.google.com/machine-learning/fairness-overview/'\u003eML fairness\u003c/a\u003e.\n",
        "\n",
        "Note: \u003ca target='_blank' href='https://www.tensorflow.org/tfx/model_analysis'\u003eTensorFlow Model Analysis\u003c/a\u003e is a powerful tool for understanding how well your model predicts for various segments of your data, including understanding how your model may reinforce societal biases and disparities."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "RptgLn2RYuK3"
      },
      "source": [
        "## Python check, imports, and globals\n",
        "First we'll make sure that we're using Python 3, and then go ahead and install and import the stuff we need."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "tFcdSuXTidhH"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "# Confirm that we're using Python 3\n",
        "assert sys.version_info.major is 3, 'Oops, not running Python 3. Use Runtime \u003e Change runtime type'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "K4QXVIM7iglN"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import pprint\n",
        "\n",
        "import tensorflow as tf\n",
        "print('TF: {}'.format(tf.__version__))\n",
        "\n",
        "print('Installing Apache Beam')\n",
        "!pip install -Uq apache_beam==2.17.0\n",
        "import apache_beam as beam\n",
        "print('Beam: {}'.format(beam.__version__))\n",
        "\n",
        "print('Installing TensorFlow Transform')\n",
        "!pip install -q tensorflow-transform==0.21\n",
        "import tensorflow_transform as tft\n",
        "print('Transform: {}'.format(tft.__version__))\n",
        "\n",
        "import tensorflow_transform.beam as tft_beam\n",
        "\n",
        "!wget https://storage.googleapis.com/artifacts.tfx-oss-public.appspot.com/datasets/census/adult.data\n",
        "!wget https://storage.googleapis.com/artifacts.tfx-oss-public.appspot.com/datasets/census/adult.test\n",
        "\n",
        "train = './adult.data'\n",
        "test = './adult.test'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "CxOxaaOYRfl7"
      },
      "source": [
        "### Name our columns\n",
        "We'll create some handy lists for referencing the columns in our dataset."
      ]<commit_msg>Fix census notebook, pin tf to 2.1<commit_after>{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "uAttKaKmT435"
      },
      "source": [
        "\u003cdiv class=\"devsite-table-wrapper\"\u003e\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "\u003ctd\u003e\u003ca target=\"_blank\" href=\"https://www.tensorflow.org/tfx/tutorials/transform/census\"\u003e\n",
        "\u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" /\u003eView on TensorFlow.org\u003c/a\u003e\u003c/td\u003e\n",
        "\u003ctd\u003e\u003ca target=\"_blank\" href=\"https://colab.sandbox.google.com/github/tensorflow/tfx/blob/master/docs/tutorials/transform/census.ipynb\"\u003e\n",
        "\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\"\u003eRun in Google Colab\u003c/a\u003e\u003c/td\u003e\n",
        "\u003ctd\u003e\u003ca target=\"_blank\" href=\"https://github.com/tensorflow/tfx/blob/master/docs/tutorials/transform/census.ipynb\"\u003e\n",
        "\u003cimg width=32px src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\"\u003eView source on GitHub\u003c/a\u003e\u003c/td\u003e\n",
        "\u003c/table\u003e\u003c/div\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "tghWegsjhpkt"
      },
      "source": [
        "##### Copyright &copy; 2020 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "rSGJWC5biBiG"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "mPt5BHTwy_0F"
      },
      "source": [
        "# Preprocessing data with TensorFlow Transform\n",
        "***The Feature Engineering Component of TensorFlow Extended (TFX)***\n",
        "\n",
        "This example colab notebook provides a somewhat more advanced example of how \u003ca target='_blank' href='https://www.tensorflow.org/tfx/transform/'\u003eTensorFlow Transform\u003c/a\u003e (`tf.Transform`) can be used to preprocess data using exactly the same code for both training a model and serving inferences in production.\n",
        "\n",
        "TensorFlow Transform is a library for preprocessing input data for TensorFlow, including creating features that require a full pass over the training dataset.  For example, using TensorFlow Transform you could:\n",
        "\n",
        "* Normalize an input value by using the mean and standard deviation\n",
        "* Convert strings to integers by generating a vocabulary over all of the input values\n",
        "* Convert floats to integers by assigning them to buckets, based on the observed data distribution\n",
        "\n",
        "TensorFlow has built-in support for manipulations on a single example or a batch of examples. `tf.Transform` extends these capabilities to support full passes over the entire training dataset.\n",
        "\n",
        "The output of `tf.Transform` is exported as a TensorFlow graph which you can use for both training and serving. Using the same graph for both training and serving can prevent skew, since the same transformations are applied in both stages.\n",
        "\n",
        "Key Point: In order to understand `tf.Transform` and how it works with Apache Beam, you'll need to know a little bit about Apache Beam itself.  The \u003ca target='_blank' href='https://beam.apache.org/documentation/programming-guide/'\u003eBeam Programming Guide\u003c/a\u003e is a great place to start."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "_tQUubddMvnP"
      },
      "source": [
        "##What we're doing in this example\n",
        "\n",
        "In this example we'll be processing a \u003ca target='_blank' href='https://archive.ics.uci.edu/ml/machine-learning-databases/adult'\u003ewidely used dataset containing census data\u003c/a\u003e, and training a model to do classification.  Along the way we'll be transforming the data using `tf.Transform`.\n",
        "\n",
        "Key Point: As a modeler and developer, think about how this data is used and the potential benefits and harm a model's predictions can cause. A model like this could reinforce societal biases and disparities. Is a feature relevant to the problem you want to solve or will it introduce bias? For more information, read about \u003ca target='_blank' href='https://developers.google.com/machine-learning/fairness-overview/'\u003eML fairness\u003c/a\u003e.\n",
        "\n",
        "Note: \u003ca target='_blank' href='https://www.tensorflow.org/tfx/model_analysis'\u003eTensorFlow Model Analysis\u003c/a\u003e is a powerful tool for understanding how well your model predicts for various segments of your data, including understanding how your model may reinforce societal biases and disparities."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "RptgLn2RYuK3"
      },
      "source": [
        "## Python check, imports, and globals\n",
        "First we'll make sure that we're using Python 3, and then go ahead and install and import the stuff we need."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "tFcdSuXTidhH"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "# Confirm that we're using Python 3\n",
        "assert sys.version_info.major is 3, 'Oops, not running Python 3. Use Runtime \u003e Change runtime type'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "K4QXVIM7iglN"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import pprint\n",
        "\n",
        "# TODO(b/156035765): update tensorflow version.\n",
        "!pip install -Uq tensorflow==2.1\n",
        "import tensorflow as tf\n",
        "print('TF: {}'.format(tf.__version__))\n",
        "\n",
        "print('Installing Apache Beam')\n",
        "!pip install -Uq apache_beam==2.20.0\n",
        "import apache_beam as beam\n",
        "print('Beam: {}'.format(beam.__version__))\n",
        "\n",
        "print('Installing TensorFlow Transform')\n",
        "!pip install -q tensorflow-transform==0.21.2\n",
        "import tensorflow_transform as tft\n",
        "print('Transform: {}'.format(tft.__version__))\n",
        "\n",
        "import tensorflow_transform.beam as tft_beam\n",
        "\n",
        "!wget https://storage.googleapis.com/artifacts.tfx-oss-public.appspot.com/datasets/census/adult.data\n",
        "!wget https://storage.googleapis.com/artifacts.tfx-oss-public.appspot.com/datasets/census/adult.test\n",
        "\n",
        "train = './adult.data'\n",
        "test = './adult.test'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "CxOxaaOYRfl7"
      },
      "source": [
        "### Name our columns\n",
        "We'll create some handy lists for referencing the columns in our dataset."
      ]
{'fraction_non_alphanumeric': 0.1408085053776734, 'fraction_numerical': 0.02781555198417604, 'mean_word_length': 2.534076015727392, 'pattern_counts': {'":': 112, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 28, 'lorem ipsum': 0, 'www.': 14, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6523985', 'n_tokens_mistral': 5430, 'n_tokens_neox': 4994, 'n_words': 1396}