<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Kubeflow pipelines\n",
    "\n",
    "**Learning Objectives:**\n",
    "  1. Learn how to deploy a Kubeflow cluster on GCP\n",
    "  1. Learn how to create a experiment in Kubeflow\n",
    "  1. Learn how to package you code into a Kubeflow pipeline\n",
    "  1. Learn how to run a Kubeflow pipeline in a repeatable and traceable way\n",
    "\n",
    "\n",
    "## Introduction\n",
    "\n",
    "In this notebook, we will first setup a Kubeflow cluster on GCP.\n",
    "Then, we will create a Kubeflow experiment and a Kubflow pipeline from our taxifare machine learning code. At last, we will run the pipeline on the Kubeflow cluster, providing us with a reproducible and traceable way to execute machine learning code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "flake8-noqa-cell"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: kfp in /home/jupyter/.local/lib/python3.7/site-packages (1.7.2)\n",
      "Requirement already satisfied: Deprecated<2,>=1.2.7 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp) (1.2.12)\n",
      "Requirement already satisfied: requests-toolbelt<1,>=0.8.0 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp) (0.9.1)\n",
      "Requirement already satisfied: jsonschema<4,>=3.0.1 in /opt/conda/lib/python3.7/site-packages (from kfp) (3.2.0)\n",
      "Requirement already satisfied: strip-hints<1,>=0.1.8 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp) (0.1.10)\n",
      "Requirement already satisfied: google-auth<2,>=1.6.1 in /opt/conda/lib/python3.7/site-packages (from kfp) (1.34.0)\n",
      "Requirement already satisfied: google-cloud-storage<2,>=1.20.0 in /opt/conda/lib/python3.7/site-packages (from kfp) (1.41.1)\n",
      "Requirement already satisfied: PyYAML<6,>=5.3 in /opt/conda/lib/python3.7/site-packages (from kfp) (5.4.1)\n",
      "Requirement already satisfied: kfp-server-api<2.0.0,>=1.1.2 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp) (1.7.0)\n",
      "Requirement already satisfied: pydantic<2,>=1.8.2 in /opt/conda/lib/python3.7/site-packages (from kfp) (1.8.2)\n",
      "Requirement already satisfied: fire<1,>=0.3.1 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp) (0.4.0)\n",
      "Requirement already satisfied: google-api-python-client<2,>=1.7.8 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp) (1.12.8)\n",
      "Requirement already satisfied: docstring-parser<1,>=0.7.3 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp) (0.10)\n",
      "Requirement already satisfied: absl-py<=0.11,>=0.9 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp) (0.11.0)\n",
      "Requirement already satisfied: kubernetes<13,>=8.0.0 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp) (12.0.1)\n",
      "Requirement already satisfied: cloudpickle<2,>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from kfp) (1.6.0)\n",
      "Requirement already satisfied: protobuf<4,>=3.13.0 in /opt/conda/lib/python3.7/site-packages (from kfp) (3.16.0)\n",
      "Requirement already satisfied: tabulate<1,>=0.8.6 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp) (0.8.9)\n",
      "Requirement already satisfied: kfp-pipeline-spec<0.2.0,>=0.1.9 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp) (0.1.9)\n",
      "Requirement already satisfied: click<8,>=7.1.1 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp) (7.1.2)\n",
      "Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from absl-py<=0.11,>=0.9->kfp) (1.16.0)\n",
      "Requirement already satisfied: wrapt<2,>=1.10 in /opt/conda/lib/python3.7/site-packages (from Deprecated<2,>=1.2.7->kfp) (1.12.1)\n",
      "Requirement already satisfied: termcolor in /opt/conda/lib/python3.7/site-packages (from fire<1,>=0.3.1->kfp) (1.1.0)\n",
      "Requirement already satisfied: google-auth-httplib2>=0.0.3 in /opt/conda/lib/python3.7/site-packages (from google-api-python-client<2,>=1.7.8->kfp) (0.1.0)\n",
      "Requirement already satisfied: google-api-core<2dev,>=1.21.0 in /opt/conda/lib/python3.7/site-packages (from google-api-python-client<2,>=1.7.8->kfp) (1.31.1)\n",
      "Requirement already satisfied: httplib2<1dev,>=0.15.0 in /opt/conda/lib/python3.7/site-packages (from google-api-python-client<2,>=1.7.8->kfp) (0.19.1)\n",
      "Requirement already satisfied: uritemplate<4dev,>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from google-api-python-client<2,>=1.7.8->kfp) (3.0.1)\n",
      "Requirement already satisfied: packaging>=14.3 in /opt/conda/lib/python3.7/site-packages (from google-api-core<2dev,>=1.21.0->google-api-python-client<2,>=1.7.8->kfp) (21.0)\n",
      "Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core<2dev,>=1.21.0->google-api-python-client<2,>=1.7.8->kfp) (1.53.0)\n",
      "Requirement already satisfied: setuptools>=40.3.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core<2dev,>=1.21.0->google-api-python-client<2,>=1.7.8->kfp) (49.6.0.post20210108)\n",
      "Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core<2dev,>=1.21.0->google-api-python-client<2,>=1.7.8->kfp) (2.25.1)\n",
      "Requirement already satisfied: pytz in /opt/conda/lib/python3.7/site-packages (from google-api-core<2dev,>=1.21.0->google-api-python-client<2,>=1.7.8->kfp) (2021.1)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.1->kfp) (0.2.7)\n",
      "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.1->kfp) (4.2.2)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.1->kfp) (4.7.2)\n",
      "Requirement already satisfied: google-resumable-media<3.0dev,>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-storage<2,>=1.20.0->kfp) (1.3.2)\n",
      "Requirement already satisfied: google-cloud-core<3.0dev,>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-storage<2,>=1.20.0->kfp) (1.7.2)\n",
      "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /opt/conda/lib/python3.7/site-packages (from google-resumable-media<3.0dev,>=1.3.0->google-cloud-storage<2,>=1.20.0->kfp) (1.1.2)\n",
      "Requirement already satisfied: cffi>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=1.3.0->google-cloud-storage<2,>=1.20.0->kfp) (1.14.6)\n",
      "Requirement already satisfied: pycparser in /opt/conda/lib/python3.7/site-packages (from cffi>=1.0.0->google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=1.3.0->google-cloud-storage<2,>=1.20.0->kfp) (2.20)\n",
      "Requirement already satisfied: pyparsing<3,>=2.4.2 in /opt/conda/lib/python3.7/site-packages (from httplib2<1dev,>=0.15.0->google-api-python-client<2,>=1.7.8->kfp) (2.4.7)\n",
      "Requirement already satisfied: attrs>=17.4.0 in /opt/conda/lib/python3.7/site-packages (from jsonschema<4,>=3.0.1->kfp) (21.2.0)\n",
      "Requirement already satisfied: importlib-metadata in /opt/conda/lib/python3.7/site-packages (from jsonschema<4,>=3.0.1->kfp) (4.6.3)\n",
      "Requirement already satisfied: pyrsistent>=0.14.0 in /opt/conda/lib/python3.7/site-packages (from jsonschema<4,>=3.0.1->kfp) (0.17.3)\n",
      "Requirement already satisfied: certifi in /opt/conda/lib/python3.7/site-packages (from kfp-server-api<2.0.0,>=1.1.2->kfp) (2021.5.30)\n",
      "Requirement already satisfied: python-dateutil in /opt/conda/lib/python3.7/site-packages (from kfp-server-api<2.0.0,>=1.1.2->kfp) (2.8.2)\n",
      "Requirement already satisfied: urllib3>=1.15 in /opt/conda/lib/python3.7/site-packages (from kfp-server-api<2.0.0,>=1.1.2->kfp) (1.26.6)\n",
      "Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from kubernetes<13,>=8.0.0->kfp) (0.57.0)\n",
      "Requirement already satisfied: requests-oauthlib in /opt/conda/lib/python3.7/site-packages (from kubernetes<13,>=8.0.0->kfp) (1.3.0)\n",
      "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /opt/conda/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.1->kfp) (0.4.8)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /opt/conda/lib/python3.7/site-packages (from pydantic<2,>=1.8.2->kfp) (3.10.0.0)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core<2dev,>=1.21.0->google-api-python-client<2,>=1.7.8->kfp) (2.10)\n",
      "Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core<2dev,>=1.21.0->google-api-python-client<2,>=1.7.8->kfp) (4.0.0)\n",
      "Requirement already satisfied: wheel in /opt/conda/lib/python3.7/site-packages (from strip-hints<1,>=0.1.8->kfp) (0.36.2)\n",
      "Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->jsonschema<4,>=3.0.1->kfp) (3.5.0)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from requests-oauthlib->kubernetes<13,>=8.0.0->kfp) (3.1.1)\n"
     ]
    }
   ],
   "source": [
    "!pip3 install --user kfp --upgrade"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hhq5zEbGg0XX"
   },
   "source": [
    "### Restart the kernel\n",
    "\n",
    "After you install the additional packages, you need to restart the notebook kernel so it can find the packages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XoEqT2Y4DJmf"
   },
   "source": [
    "### Import libraries and define constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "flake8-noqa-cell"
    ]
   },
   "outputs": [],
   "source": [
    "from os import path\n",
    "\n",
    "import kfp\n",
    "import kfp.compiler as compiler\n",
    "import kfp.components as comp\n",
    "import kfp.dsl as dsl\n",
    "import kfp.gcp as gcp\n",
    "import kfp.notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup a Kubeflow cluster on GCP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO 1**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To deploy a [Kubeflow](https://www.kubeflow.org/) cluster\n",
    "in your GCP project, use the [AI Platform pipelines](https://console.cloud.google.com/ai-platform/pipelines):\n",
    "\n",
    "1. Go to [AI Platform Pipelines](https://console.cloud.google.com/ai-platform/pipelines) in the GCP Console.\n",
    "1. Create a new instance\n",
    "2. Hit \"Configure\"\n",
    "3. Check the box \"Allow access to the following Cloud APIs\"\n",
    "1. Hit \"Create Cluster\"\n",
    "4. Hit \"Deploy\"\n",
    "\n",
    "When the cluster is ready, go back to the AI Platform pipelines page and click on \"SETTINGS\" entry for your cluster.\n",
    "This will bring up a pop up with code snippets on how to access the cluster \n",
    "programmatically. \n",
    "\n",
    "Copy the \"host\" entry and set the \"HOST\" variable below with that.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "HOST = \"\"  # None  # TODO: fill in the HOST information for the cluster"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Authenticate your KFP cluster with a  Kubernetes secret\n",
    "\n",
    "If you run pipelines that requires calling any GCP services, you need to set the application default credential to a pipeline step by mounting the proper GCP service account token as a Kubernetes secret.\n",
    "\n",
    "First point your kubectl current context to your cluster. Go back to your [Kubeflow cluster dashboard](https://console.cloud.google.com/ai-platform/pipelines/clusters) or navigate to `Navigation menu > AI Platform > Pipelines` and look to see the cluster name, zone and namespace for the pipeline you deployed above. It's likely called `cluster-1` if this is the first AI Pipelines you've created. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [<commit_msg>clear up placeholder for host variable<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Kubeflow pipelines\n",
    "\n",
    "**Learning Objectives:**\n",
    "  1. Learn how to deploy a Kubeflow cluster on GCP\n",
    "  1. Learn how to create a experiment in Kubeflow\n",
    "  1. Learn how to package you code into a Kubeflow pipeline\n",
    "  1. Learn how to run a Kubeflow pipeline in a repeatable and traceable way\n",
    "\n",
    "\n",
    "## Introduction\n",
    "\n",
    "In this notebook, we will first setup a Kubeflow cluster on GCP.\n",
    "Then, we will create a Kubeflow experiment and a Kubflow pipeline from our taxifare machine learning code. At last, we will run the pipeline on the Kubeflow cluster, providing us with a reproducible and traceable way to execute machine learning code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "flake8-noqa-cell"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: kfp in /home/jupyter/.local/lib/python3.7/site-packages (1.7.2)\n",
      "Requirement already satisfied: Deprecated<2,>=1.2.7 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp) (1.2.12)\n",
      "Requirement already satisfied: requests-toolbelt<1,>=0.8.0 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp) (0.9.1)\n",
      "Requirement already satisfied: jsonschema<4,>=3.0.1 in /opt/conda/lib/python3.7/site-packages (from kfp) (3.2.0)\n",
      "Requirement already satisfied: strip-hints<1,>=0.1.8 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp) (0.1.10)\n",
      "Requirement already satisfied: google-auth<2,>=1.6.1 in /opt/conda/lib/python3.7/site-packages (from kfp) (1.34.0)\n",
      "Requirement already satisfied: google-cloud-storage<2,>=1.20.0 in /opt/conda/lib/python3.7/site-packages (from kfp) (1.41.1)\n",
      "Requirement already satisfied: PyYAML<6,>=5.3 in /opt/conda/lib/python3.7/site-packages (from kfp) (5.4.1)\n",
      "Requirement already satisfied: kfp-server-api<2.0.0,>=1.1.2 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp) (1.7.0)\n",
      "Requirement already satisfied: pydantic<2,>=1.8.2 in /opt/conda/lib/python3.7/site-packages (from kfp) (1.8.2)\n",
      "Requirement already satisfied: fire<1,>=0.3.1 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp) (0.4.0)\n",
      "Requirement already satisfied: google-api-python-client<2,>=1.7.8 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp) (1.12.8)\n",
      "Requirement already satisfied: docstring-parser<1,>=0.7.3 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp) (0.10)\n",
      "Requirement already satisfied: absl-py<=0.11,>=0.9 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp) (0.11.0)\n",
      "Requirement already satisfied: kubernetes<13,>=8.0.0 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp) (12.0.1)\n",
      "Requirement already satisfied: cloudpickle<2,>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from kfp) (1.6.0)\n",
      "Requirement already satisfied: protobuf<4,>=3.13.0 in /opt/conda/lib/python3.7/site-packages (from kfp) (3.16.0)\n",
      "Requirement already satisfied: tabulate<1,>=0.8.6 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp) (0.8.9)\n",
      "Requirement already satisfied: kfp-pipeline-spec<0.2.0,>=0.1.9 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp) (0.1.9)\n",
      "Requirement already satisfied: click<8,>=7.1.1 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp) (7.1.2)\n",
      "Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from absl-py<=0.11,>=0.9->kfp) (1.16.0)\n",
      "Requirement already satisfied: wrapt<2,>=1.10 in /opt/conda/lib/python3.7/site-packages (from Deprecated<2,>=1.2.7->kfp) (1.12.1)\n",
      "Requirement already satisfied: termcolor in /opt/conda/lib/python3.7/site-packages (from fire<1,>=0.3.1->kfp) (1.1.0)\n",
      "Requirement already satisfied: google-auth-httplib2>=0.0.3 in /opt/conda/lib/python3.7/site-packages (from google-api-python-client<2,>=1.7.8->kfp) (0.1.0)\n",
      "Requirement already satisfied: google-api-core<2dev,>=1.21.0 in /opt/conda/lib/python3.7/site-packages (from google-api-python-client<2,>=1.7.8->kfp) (1.31.1)\n",
      "Requirement already satisfied: httplib2<1dev,>=0.15.0 in /opt/conda/lib/python3.7/site-packages (from google-api-python-client<2,>=1.7.8->kfp) (0.19.1)\n",
      "Requirement already satisfied: uritemplate<4dev,>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from google-api-python-client<2,>=1.7.8->kfp) (3.0.1)\n",
      "Requirement already satisfied: packaging>=14.3 in /opt/conda/lib/python3.7/site-packages (from google-api-core<2dev,>=1.21.0->google-api-python-client<2,>=1.7.8->kfp) (21.0)\n",
      "Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core<2dev,>=1.21.0->google-api-python-client<2,>=1.7.8->kfp) (1.53.0)\n",
      "Requirement already satisfied: setuptools>=40.3.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core<2dev,>=1.21.0->google-api-python-client<2,>=1.7.8->kfp) (49.6.0.post20210108)\n",
      "Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core<2dev,>=1.21.0->google-api-python-client<2,>=1.7.8->kfp) (2.25.1)\n",
      "Requirement already satisfied: pytz in /opt/conda/lib/python3.7/site-packages (from google-api-core<2dev,>=1.21.0->google-api-python-client<2,>=1.7.8->kfp) (2021.1)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.1->kfp) (0.2.7)\n",
      "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.1->kfp) (4.2.2)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.1->kfp) (4.7.2)\n",
      "Requirement already satisfied: google-resumable-media<3.0dev,>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-storage<2,>=1.20.0->kfp) (1.3.2)\n",
      "Requirement already satisfied: google-cloud-core<3.0dev,>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-storage<2,>=1.20.0->kfp) (1.7.2)\n",
      "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /opt/conda/lib/python3.7/site-packages (from google-resumable-media<3.0dev,>=1.3.0->google-cloud-storage<2,>=1.20.0->kfp) (1.1.2)\n",
      "Requirement already satisfied: cffi>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=1.3.0->google-cloud-storage<2,>=1.20.0->kfp) (1.14.6)\n",
      "Requirement already satisfied: pycparser in /opt/conda/lib/python3.7/site-packages (from cffi>=1.0.0->google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=1.3.0->google-cloud-storage<2,>=1.20.0->kfp) (2.20)\n",
      "Requirement already satisfied: pyparsing<3,>=2.4.2 in /opt/conda/lib/python3.7/site-packages (from httplib2<1dev,>=0.15.0->google-api-python-client<2,>=1.7.8->kfp) (2.4.7)\n",
      "Requirement already satisfied: attrs>=17.4.0 in /opt/conda/lib/python3.7/site-packages (from jsonschema<4,>=3.0.1->kfp) (21.2.0)\n",
      "Requirement already satisfied: importlib-metadata in /opt/conda/lib/python3.7/site-packages (from jsonschema<4,>=3.0.1->kfp) (4.6.3)\n",
      "Requirement already satisfied: pyrsistent>=0.14.0 in /opt/conda/lib/python3.7/site-packages (from jsonschema<4,>=3.0.1->kfp) (0.17.3)\n",
      "Requirement already satisfied: certifi in /opt/conda/lib/python3.7/site-packages (from kfp-server-api<2.0.0,>=1.1.2->kfp) (2021.5.30)\n",
      "Requirement already satisfied: python-dateutil in /opt/conda/lib/python3.7/site-packages (from kfp-server-api<2.0.0,>=1.1.2->kfp) (2.8.2)\n",
      "Requirement already satisfied: urllib3>=1.15 in /opt/conda/lib/python3.7/site-packages (from kfp-server-api<2.0.0,>=1.1.2->kfp) (1.26.6)\n",
      "Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from kubernetes<13,>=8.0.0->kfp) (0.57.0)\n",
      "Requirement already satisfied: requests-oauthlib in /opt/conda/lib/python3.7/site-packages (from kubernetes<13,>=8.0.0->kfp) (1.3.0)\n",
      "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /opt/conda/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.1->kfp) (0.4.8)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /opt/conda/lib/python3.7/site-packages (from pydantic<2,>=1.8.2->kfp) (3.10.0.0)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core<2dev,>=1.21.0->google-api-python-client<2,>=1.7.8->kfp) (2.10)\n",
      "Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core<2dev,>=1.21.0->google-api-python-client<2,>=1.7.8->kfp) (4.0.0)\n",
      "Requirement already satisfied: wheel in /opt/conda/lib/python3.7/site-packages (from strip-hints<1,>=0.1.8->kfp) (0.36.2)\n",
      "Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->jsonschema<4,>=3.0.1->kfp) (3.5.0)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from requests-oauthlib->kubernetes<13,>=8.0.0->kfp) (3.1.1)\n"
     ]
    }
   ],
   "source": [
    "!pip3 install --user kfp --upgrade"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hhq5zEbGg0XX"
   },
   "source": [
    "### Restart the kernel\n",
    "\n",
    "After you install the additional packages, you need to restart the notebook kernel so it can find the packages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XoEqT2Y4DJmf"
   },
   "source": [
    "### Import libraries and define constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "flake8-noqa-cell"
    ]
   },
   "outputs": [],
   "source": [
    "from os import path\n",
    "\n",
    "import kfp\n",
    "import kfp.compiler as compiler\n",
    "import kfp.components as comp\n",
    "import kfp.dsl as dsl\n",
    "import kfp.gcp as gcp\n",
    "import kfp.notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup a Kubeflow cluster on GCP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO 1**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To deploy a [Kubeflow](https://www.kubeflow.org/) cluster\n",
    "in your GCP project, use the [AI Platform pipelines](https://console.cloud.google.com/ai-platform/pipelines):\n",
    "\n",
    "1. Go to [AI Platform Pipelines](https://console.cloud.google.com/ai-platform/pipelines) in the GCP Console.\n",
    "1. Create a new instance\n",
    "2. Hit \"Configure\"\n",
    "3. Check the box \"Allow access to the following Cloud APIs\"\n",
    "1. Hit \"Create Cluster\"\n",
    "4. Hit \"Deploy\"\n",
    "\n",
    "When the cluster is ready, go back to the AI Platform pipelines page and click on \"SETTINGS\" entry for your cluster.\n",
    "This will bring up a pop up with code snippets on how to access the cluster \n",
    "programmatically. \n",
    "\n",
    "Copy the \"host\" entry and set the \"HOST\" variable below with that.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "HOST = \"\"  # TODO: fill in the HOST information for the cluster"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Authenticate your KFP cluster with a  Kubernetes secret\n",
    "\n",
    "If you run pipelines that requires calling any GCP services, you need to set the application default credential to a pipeline step by mounting the proper GCP service account token as a Kubernetes secret.\n",
    "\n",
    "First point your kubectl current context to your cluster. Go back to your [Kubeflow cluster dashboard](https://console.cloud.google.com/ai-platform/pipelines/clusters) or navigate to `Navigation menu > AI Platform > Pipelines` and look to see the cluster name, zone and namespace for the pipeline you deployed above. It's likely called `cluster-1` if this is the first AI Pipelines you've created. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
{'fraction_non_alphanumeric': 0.19488518966122179, 'fraction_numerical': 0.062297766168960274, 'mean_word_length': 3.3455869896662715, 'pattern_counts': {'":': 96, '<': 173, '<?xml version=': 0, '>': 315, 'https://': 8, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4232997', 'n_tokens_mistral': 10385, 'n_tokens_neox': 10135, 'n_words': 1918}