<commit_before>{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "wdeKOEkv1Fe8"
      },
      "source": [
        "##### Copyright \u0026copy; 2020 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "c2jyGuiG1gHr"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "23R0Z9RojXYW"
      },
      "source": [
        "# TFX Keras Component Tutorial\n",
        "\n",
        "***A Component-by-Component Introduction to TensorFlow Extended (TFX)***"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "LidV2qsXm4XC"
      },
      "source": [
        "Note: We recommend running this tutorial in a Colab notebook, with no setup required!  Just click \"Run in Google Colab\".\n",
        "\n",
        "\u003cdiv class=\"devsite-table-wrapper\"\u003e\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "\u003ctd\u003e\u003ca target=\"_blank\" href=\"https://www.tensorflow.org/tfx/tutorials/tfx/components_keras\"\u003e\n",
        "\u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" /\u003eView on TensorFlow.org\u003c/a\u003e\u003c/td\u003e\n",
        "\u003ctd\u003e\u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tfx/blob/master/docs/tutorials/tfx/components_keras.ipynb\"\u003e\n",
        "\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\"\u003eRun in Google Colab\u003c/a\u003e\u003c/td\u003e\n",
        "\u003ctd\u003e\u003ca target=\"_blank\" href=\"https://github.com/tensorflow/tfx/tree/master/docs/tutorials/tfx/components_keras.ipynb\"\u003e\n",
        "\u003cimg width=32px src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\"\u003eView source on GitHub\u003c/a\u003e\u003c/td\u003e\n",
        "\u003c/table\u003e\u003c/div\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "KAD1tLoTm_QS"
      },
      "source": [
        "\n",
        "This Colab-based tutorial will interactively walk through each built-in component of TensorFlow Extended (TFX).\n",
        "\n",
        "It covers every step in an end-to-end machine learning pipeline, from data ingestion to pushing a model to serving.\n",
        "\n",
        "When you're done, the contents of this notebook can be automatically exported as TFX pipeline source code, which you can orchestrate with Apache Airflow and Apache Beam.\n",
        "\n",
        "Note: This notebook demonstrates the use of native Keras models in TFX pipelines. **TFX only supports the TensorFlow 2 version of Keras**.\n",
        "\n",
        "Note: This notebook and its associated APIs are **experimental** and are\n",
        "in active development.  Major changes in functionality, behavior, and\n",
        "presentation are expected."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "sfSQ-kX-MLEr"
      },
      "source": [
        "## Background\n",
        "This notebook demonstrates how to use TFX in a Jupyter/Colab environment.  Here, we walk through the Chicago Taxi example in an interactive notebook.\n",
        "\n",
        "Working in an interactive notebook is a useful way to become familiar with the structure of a TFX pipeline.  It's also useful when doing development of your own pipelines as a lightweight development environment, but you should be aware that there are differences in the way interactive notebooks are orchestrated, and how they access metadata artifacts.\n",
        "\n",
        "### Orchestration\n",
        "\n",
        "In a production deployment of TFX, you will use an orchestrator such as Apache Airflow, Kubeflow Pipelines, or Apache Beam to orchestrate a pre-defined pipeline graph of TFX components.  In an interactive notebook, the notebook itself is the orchestrator, running each TFX component as you execute the notebook cells.\n",
        "\n",
        "### Metadata\n",
        "\n",
        "In a production deployment of TFX, you will access metadata through the ML Metadata (MLMD) API.  MLMD stores metadata properties in a database such as MySQL or SQLite, and stores the metadata payloads in a persistent store such as on your filesystem.  In an interactive notebook, both properties and payloads are stored in an ephemeral SQLite database in the `/tmp` directory on the Jupyter notebook or Colab server."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "2GivNBNYjb3b"
      },
      "source": [
        "## Setup\n",
        "First, we install and import the necessary packages, set up paths, and download data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "MZOYTt1RW4TK"
      },
      "source": [
        "### Install TFX\n",
        "\n",
        "**Note: In Google Colab, because of package updates, the first time you run this cell you must restart the runtime (Runtime \u003e Restart runtime ...).**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "S4SQA7Q5nej3"
      },<commit_msg>Updating pip before installing TFX<commit_after>{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "wdeKOEkv1Fe8"
      },
      "source": [
        "##### Copyright \u0026copy; 2020 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "c2jyGuiG1gHr"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "23R0Z9RojXYW"
      },
      "source": [
        "# TFX Keras Component Tutorial\n",
        "\n",
        "***A Component-by-Component Introduction to TensorFlow Extended (TFX)***"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "LidV2qsXm4XC"
      },
      "source": [
        "Note: We recommend running this tutorial in a Colab notebook, with no setup required!  Just click \"Run in Google Colab\".\n",
        "\n",
        "\u003cdiv class=\"devsite-table-wrapper\"\u003e\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "\u003ctd\u003e\u003ca target=\"_blank\" href=\"https://www.tensorflow.org/tfx/tutorials/tfx/components_keras\"\u003e\n",
        "\u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" /\u003eView on TensorFlow.org\u003c/a\u003e\u003c/td\u003e\n",
        "\u003ctd\u003e\u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tfx/blob/master/docs/tutorials/tfx/components_keras.ipynb\"\u003e\n",
        "\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\"\u003eRun in Google Colab\u003c/a\u003e\u003c/td\u003e\n",
        "\u003ctd\u003e\u003ca target=\"_blank\" href=\"https://github.com/tensorflow/tfx/tree/master/docs/tutorials/tfx/components_keras.ipynb\"\u003e\n",
        "\u003cimg width=32px src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\"\u003eView source on GitHub\u003c/a\u003e\u003c/td\u003e\n",
        "\u003c/table\u003e\u003c/div\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "KAD1tLoTm_QS"
      },
      "source": [
        "\n",
        "This Colab-based tutorial will interactively walk through each built-in component of TensorFlow Extended (TFX).\n",
        "\n",
        "It covers every step in an end-to-end machine learning pipeline, from data ingestion to pushing a model to serving.\n",
        "\n",
        "When you're done, the contents of this notebook can be automatically exported as TFX pipeline source code, which you can orchestrate with Apache Airflow and Apache Beam.\n",
        "\n",
        "Note: This notebook demonstrates the use of native Keras models in TFX pipelines. **TFX only supports the TensorFlow 2 version of Keras**.\n",
        "\n",
        "Note: This notebook and its associated APIs are **experimental** and are\n",
        "in active development.  Major changes in functionality, behavior, and\n",
        "presentation are expected."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "sfSQ-kX-MLEr"
      },
      "source": [
        "## Background\n",
        "This notebook demonstrates how to use TFX in a Jupyter/Colab environment.  Here, we walk through the Chicago Taxi example in an interactive notebook.\n",
        "\n",
        "Working in an interactive notebook is a useful way to become familiar with the structure of a TFX pipeline.  It's also useful when doing development of your own pipelines as a lightweight development environment, but you should be aware that there are differences in the way interactive notebooks are orchestrated, and how they access metadata artifacts.\n",
        "\n",
        "### Orchestration\n",
        "\n",
        "In a production deployment of TFX, you will use an orchestrator such as Apache Airflow, Kubeflow Pipelines, or Apache Beam to orchestrate a pre-defined pipeline graph of TFX components.  In an interactive notebook, the notebook itself is the orchestrator, running each TFX component as you execute the notebook cells.\n",
        "\n",
        "### Metadata\n",
        "\n",
        "In a production deployment of TFX, you will access metadata through the ML Metadata (MLMD) API.  MLMD stores metadata properties in a database such as MySQL or SQLite, and stores the metadata payloads in a persistent store such as on your filesystem.  In an interactive notebook, both properties and payloads are stored in an ephemeral SQLite database in the `/tmp` directory on the Jupyter notebook or Colab server."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "2GivNBNYjb3b"
      },
      "source": [
        "## Setup\n",
        "First, we install and import the necessary packages, set up paths, and download data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Fmgi8ZvQkScg"
      },
      "source": [
        "### Upgrade Pip"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "as4OTe2ukSqm"
      },
      "outputs": [],
      "source": [
        "try:\n",
        "  import colab\n",
        "  !pip install --upgrade pip\n",
        "except:\n",
        "  pass"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "MZOYTt1RW4TK"
      },
      "source": [
        "### Install TFX\n",
        "\n",
        "**Note: In Google Colab, because of package updates, the first time you run this cell you must restart the runtime (Runtime \u003e Restart runtime ...).**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "S4SQA7Q5nej3"
      },
{'fraction_non_alphanumeric': 0.1363050564779016, 'fraction_numerical': 0.02441058297323933, 'mean_word_length': 2.357597353016035, 'pattern_counts': {'":': 115, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 14, 'lorem ipsum': 0, 'www.': 10, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2221991', 'n_tokens_mistral': 4368, 'n_tokens_neox': 4095, 'n_words': 1211}