<commit_before>{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "copyright"
      },
      "outputs": [],
      "source": [
        "# Copyright 2021 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "title:generic,gcp"
      },
      "source": [
        "# E2E ML on GCP: MLOps stage 3 : formalization: get started with AutoML pipeline components\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td>\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_automl_pipeline_components.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
        "      View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_automl_pipeline_components.ipynb\">\n",
        "      Open in Google Cloud Notebooks\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "<br/><br/><br/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "overview:mlops"
      },
      "source": [
        "## Overview\n",
        "\n",
        "\n",
        "This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with AutoML pipeline components."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dataset:flowers,icn"
      },
      "source": [
        "### Dataset\n",
        "\n",
        "The dataset used for this tutorial is the [Flowers dataset](https://www.tensorflow.org/datasets/catalog/tf_flowers) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of flower an image is from a class of five flowers: daisy, dandelion, rose, sunflower, or tulip."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "objective:mlops,stage3,get_started_automl_pipeline_components"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you learn how to use prebuilt `Google Cloud Pipeline Components` for `Vertex AI AutoML`.\n",
        "\n",
        "This tutorial uses the following Google Cloud ML services:\n",
        "\n",
        "- `Vertex AI Pipelines`\n",
        "- `Vertex AI AutoML`\n",
        "- `Google Cloud Pipeline Components`\n",
        "- `Vertex AI Dataset, Model and Endpoint` resources\n",
        "- `Vertex AI Prediction`\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "- Construct a pipeline for:\n",
        "    - Training a Vertex AI AutoML trained model.\n",
        "    - Test the serving binary with a batch prediction job.\n",
        "    - Deploying a Vertex AI AutoML trained model.\n",
        "- Execute a Vertex AI pipeline."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "install_mlops"
      },
      "source": [
        "## Installations\n",
        "\n",
        "Install *one time* the packages for executing the MLOps notebooks."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "install_mlops"
      },
      "outputs": [],
      "source": [
        "ONCE_ONLY = False\n",
        "if ONCE_ONLY:\n",
        "    ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
        "    ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
        "    ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
        "    ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
        "    ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
        "    ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
        "    ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
        "    ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
        "    ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
        "    ! pip3 install --upgrade pyarrow $USER_FLAG\n",
        "    ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
        "    ! pip3 install --upgrade kfp $USER_FLAG\n",
        "    ! pip3 install --upgrade torchvision $USER_FLAG\n",
        "    ! pip3 install --upgrade rpy2 $USER_FLAG"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "restart"
      },
      "source": [
        "### Restart the kernel\n",
        "\n",
        "Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "restart"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "if not os.getenv(\"IS_TESTING\"):\n",
        "    # Automatically restart kernel after installs\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "project_id"
      },
      "source": [
        "#### Set your project ID\n",
        "\n",
        "**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_project_id"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "autoset_project_id"
      },
      "outputs": [],
      "source": [
        "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
        "    # Get your GCP project id from gcloud\n",
        "    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
        "    PROJECT_ID = shell_output[0]\n",
        "    print(\"Project ID:\", PROJECT_ID)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_gcloud_project_id"
      },
      "outputs": [],
      "source": [
        "! gcloud config set project $PROJECT_ID"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "region"
      },
      "source": [
        "#### Region\n",
        "\n",
        "You can also change the `REGION` variable, which is used for operations\n",
        "throughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
        "\n",
        "- Americas: `us-central1`\n",
        "- Europe: `europe-west4`\n",
        "- Asia Pacific: `asia-east1`\n",
        "\n",
        "You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
        "\n",
        "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "region"
      },
      "outputs": [],
      "source": [
        "REGION = \"us-central1\"  # @param {type: \"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "timestamp"
      },
      "source": [
        "#### Timestamp\n",
        "\n",
        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "timestamp"
      },
      "outputs": [],
      "source": [
        "from datetime import datetime\n",
        "\n",
        "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bucket:mbsdk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "When you initialize the Vertex SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
        "\n",
        "Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bucket"
      },
      "outputs": [],
      "source": [
        "BUCKET_NAME = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "autoset_bucket"
      },
      "outputs": [],
      "source": [
        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
        "    BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_bucket"
      },
      "source": [
        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_bucket"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l $REGION $BUCKET_NAME"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "validate_bucket"
      },
      "source": [
        "Finally, validate access to your Cloud Storage bucket by examining its contents:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "validate_bucket"
      },
      "outputs": [],
      "source": [
        "! gsutil ls -al $BUCKET_NAME"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "set_service_account"
      },
      "source": [
        "#### Service Account\n",
        "\n",
        "**If you don't know your service account**, try to get your service account using `gcloud` command by executing the second cell below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_service_account"
      },
      "outputs": [],
      "source": [
        "SERVICE_ACCOUNT = \"[your-service-account]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "autoset_service_account"
      },
      "outputs": [],
      "source": [
        "if (\n",
        "    SERVICE_ACCOUNT == \"\"\n",
        "    or SERVICE_ACCOUNT is None\n",
        "    or SERVICE_ACCOUNT == \"[your-service-account]\"\n",
        "):\n",
        "    # Get your GCP project id from gcloud\n",
        "    shell_output = !gcloud auth list 2>/dev/null\n",
        "    SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n",
        "    print(\"Service Account:\", SERVICE_ACCOUNT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "set_service_account:pipelines"
      },
      "source": [
        "#### Set service account access for Vertex AI Pipelines\n",
        "\n",
        "Run the following commands to grant your service account access to read and write pipeline artifacts in the bucket that you created in the previous step -- you only need to run these once per service account."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_service_account:pipelines"
      },
      "outputs": [],
      "source": [
        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_NAME\n",
        "\n",
        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_NAME"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "setup_vars"
      },
      "source": [
        "### Set up variables\n",
        "\n",
        "Next, set up some variables used throughout the tutorial.\n",
        "### Import libraries and define constants"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "import_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "import google.cloud.aiplatform as aip"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "import_tf"
      },
      "source": [
        "#### Import TensorFlow\n",
        "\n",
        "Import the TensorFlow package into your Python environment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "import_tf"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "import_kfp"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "\n",
        "from kfp import dsl\n",
        "from kfp.v2 import compiler\n",
        "from kfp.v2.dsl import component"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "init_aip:mbsdk,all"
      },
      "source": [
        "### Initialize Vertex AI SDK for Python\n",
        "\n",
        "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "init_aip:mbsdk,all"
      },
      "outputs": [],
      "source": [
        "aip.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_NAME)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "import_file:u_dataset,csv"
      },
      "source": [
        "#### Location of Cloud Storage training data.\n",
        "\n",
        "Now set the variable `IMPORT_FILE` to the location of the CSV index file in Cloud Storage."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "import_file:flowers,csv,icn"
      },
      "outputs": [],
      "source": [
        "IMPORT_FILE = (\n",
        "    \"gs://cloud-samples-data/vision/automl_classification/flowers/all_data_v2.csv\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_automl_eval_component"
      },
      "source": [
        "### Create AutoML model evaluation component\n",
        "\n",
        "The Vertex AI pre-built pipeline components does not currently have a component for retrieiving the model evaluations for a AutoML model. So, you will first write your own component, as follows:\n",
        "\n",
        "- Takes as input the region and Model artifacts returned from an AutoML training component.\n",
        "- Create a client interface to the Vertex AI Model service (`metadata[\"resource_name\"]).\n",
        "- Construct the resource ID for the model from the model artifact parameter.\n",
        "- Retrieve the model evaluation\n",
        "- Return the model evaluation as a string."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_automl_eval_component"
      },
      "outputs": [],
      "source": [
        "from kfp.v2.dsl import Artifact, Input, Model\n",
        "\n",
        "\n",
        "@component(packages_to_install=[\"google-cloud-aiplatform\"])\n",
        "def evaluateAutoMLModelOp(model: Input[Artifact], region: str) -> str:\n",
        "    import logging\n",
        "\n",
        "    import google.cloud.aiplatform.gapic as gapic\n",
        "\n",
        "    # Get a reference to the Model Service client\n",
        "    client_options = {\"api_endpoint\": f\"{region}-aiplatform.googleapis.com\"}\n",
        "    model_service_client = gapic.ModelServiceClient(client_options=client_options)\n",
        "\n",
        "    model_id = model.metadata[\"resourceName\"]\n",
        "\n",
        "    model_evaluations = model_service_client.list_model_evaluations(parent=model_id)\n",
        "    model_evaluation = list(model_evaluations)[0]\n",
        "    logging.info(model_evaluation)\n",
        "    return str(model_evaluation)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_automl_pipeline:icn"
      },
      "source": [
        "## Construct AutoML training pipeline\n",
        "\n",
        "In the example below, you construct a pipeline for training an AutoML model using pre-built Google Cloud Pipeline Components for AutoML, as follows:\n",
        "\n",
        "1. Use the prebuilt component `ImageDatasetCreateOp` to create a Vertex AI Dataset resource, where:\n",
        "    - The display name for the dataset is passed into the pipeline.\n",
        "    - The import file for the dataset is passed into the pipeline.\n",
        "    - The component returns the dataset resource as `outputs[\"dataset\"]`\n",
        "\n",
        "\n",
        "2. Use the prebuilt component `AutoMLImageTrainingJobRunOp` to train a Vertex AI AutoML Model resource, where:\n",
        "    - The display name for the dataset is passed into the pipeline.\n",
        "    - The dataset is the output from the `ImageDatasetCreateOp`.\n",
        "    - The component returns the model resource as `outputs[\"model\"]`.\n",
        "\n",
        "\n",
        "3. Use the prebuild component `ModelBatchPredictOp` to do a test batch prediction, where:\n",
        "    - The model is the output from the `AutoMLTrainingJobRunOp`.\n",
        "\n",
        "\n",
        "4. Use the prebuilt component `EndpointCreateOp` to create a Vertex AI Endpoint to deploy the trained model to, where:\n",
        "    - Since the component has no dependencies on other components, by default it would be executed in parallel with the model training.\n",
        "    - The `after(training_op)` is added to serialize its execution, so its only executed if the training operation completes successfully.\n",
        "    - The component returns the endpoint resource as `outputs[\"endpoint\"]`.\n",
        "\n",
        "\n",
        "5. Use the prebuilt component `ModelDeployOp` to deploy the trained AutoML model to, where:\n",
        "    - The display name for the dataset is passed into the pipeline.\n",
        "    - The model is the output from the `AutoMLTrainingJobRunOp`.\n",
        "    - The endpoint is the output from the `EndpointCreateOp`\n",
        "\n",
        "*Note:* Since each component is executed as a graph node in its own execution context, you pass the parameter `project` for each component op, in constrast to doing a `aip.init(project=project)` if this was a Python script calling the SDK methods directly within the same execution context."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_automl_pipeline:icn"
      },
      "outputs": [],
      "source": [
        "from google_cloud_pipeline_components import aiplatform as gcc_aip\n",
        "\n",
        "PIPELINE_ROOT = \"{}/pipeline_root/automl_icn_training\".format(BUCKET_NAME)\n",
        "DEPLOY_COMPUTE = \"n1-standard-4\"\n",
        "\n",
        "\n",
        "@dsl.pipeline(\n",
        "    name=\"automl-icn-training\", description=\"AutoML image classification training\"\n",
        ")\n",
        "def pipeline(\n",
        "    import_file: str,\n",
        "    batch_files: list,\n",
        "    display_name: str,\n",
        "    bucket: str = PIPELINE_ROOT,\n",
        "    project: str = PROJECT_ID,\n",
        "    region: str = REGION,\n",
        "):\n",
        "\n",
        "    dataset_op = gcc_aip.ImageDatasetCreateOp(\n",
        "        project=project,\n",
        "        display_name=display_name,\n",
        "        gcs_source=import_file,\n",
        "        import_schema_uri=aip.schema.dataset.ioformat.image.single_label_classification,\n",
        "    )\n",
        "\n",
        "    training_op = gcc_aip.AutoMLImageTrainingJobRunOp(\n",
        "        project=project,\n",
        "        display_name=display_name,\n",
        "        prediction_type=\"classification\",\n",
        "        model_type=\"CLOUD\",\n",
        "        base_model=None,\n",
        "        dataset=dataset_op.outputs[\"dataset\"],\n",
        "        model_display_name=display_name,\n",
        "        training_fraction_split=0.6,\n",
        "        validation_fraction_split=0.2,\n",
        "        test_fraction_split=0.2,\n",
        "        budget_milli_node_hours=8000,\n",
        "    )\n",
        "\n",
        "    eval_op = evaluateAutoMLModelOp(model=training_op.outputs[\"model\"], region=region)\n",
        "\n",
        "    batch_op = gcc_aip.ModelBatchPredictOp(\n",
        "        project=project,\n",
        "        job_display_name=\"batch_predict_job\",\n",
        "        model=training_op.outputs[\"model\"],\n",
        "        gcs_source_uris=batch_files,\n",
        "        gcs_destination_output_uri_prefix=bucket,\n",
        "        instances_format=\"jsonl\",\n",
        "        predictions_format=\"jsonl\",\n",
        "        model_parameters={},\n",
        "        machine_type=DEPLOY_COMPUTE,\n",
        "        starting_replica_count=1,\n",
        "        max_replica_count=1,\n",
        "    ).after(eval_op)\n",
        "\n",
        "    endpoint_op = gcc_aip.EndpointCreateOp(\n",
        "        project=project,\n",
        "        location=region,\n",
        "        display_name=display_name,\n",
        "    ).after(batch_op)\n",
        "\n",
        "    deploy_op = gcc_aip.ModelDeployOp(\n",
        "        model=training_op.outputs[\"model\"],\n",
        "        endpoint=endpoint_op.outputs[\"endpoint\"],\n",
        "        automatic_resources_min_replica_count=1,\n",
        "        automatic_resources_max_replica_count=1,\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "get_test_items:batch_prediction"
      },
      "source": [
        "### Get test item(s)\n",
        "\n",
        "Now do a batch prediction to your Vertex model. You will use arbitrary examples out of the dataset as a test items. Don't be concerned that the examples were likely used in training the model -- we just want to demonstrate how to make a prediction."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "get_test_items:automl,icn,csv"
      },
      "outputs": [],
      "source": [
        "test_items = !gsutil cat $IMPORT_FILE | head -n2\n",
        "if len(str(test_items[0]).split(\",\")) == 3:\n",
        "    _, test_item_1, test_label_1 = str(test_items[0]).split(\",\")\n",
        "    _, test_item_2, test_label_2 = str(test_items[1]).split(\",\")\n",
        "else:\n",
        "    test_item_1, test_label_1 = str(test_items[0]).split(\",\")\n",
        "    test_item_2, test_label_2 = str(test_items[1]).split(\",\")\n",
        "\n",
        "print(test_item_1, test_label_1)\n",
        "print(test_item_2, test_label_2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "copy_test_items:batch_prediction"
      },
      "source": [
        "### Copy test item(s)\n",
        "\n",
        "For the batch prediction, copy the test items over to your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "copy_test_items:batch_prediction"
      },
      "outputs": [],
      "source": [
        "file_1 = test_item_1.split(\"/\")[-1]\n",
        "file_2 = test_item_2.split(\"/\")[-1]\n",
        "\n",
        "! gsutil cp $test_item_1 $BUCKET_NAME/$file_1\n",
        "! gsutil cp $test_item_2 $BUCKET_NAME/$file_2\n",
        "\n",
        "test_item_1 = BUCKET_NAME + \"/\" + file_1\n",
        "test_item_2 = BUCKET_NAME + \"/\" + file_2"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "make_batch_file:automl,image"
      },
      "source": [
        "### Make the batch input file\n",
        "\n",
        "Now make a batch input file, which you will store in your local Cloud Storage bucket. The batch input file can be either CSV or JSONL. You will use JSONL in this tutorial. For JSONL file, you make one dictionary entry per line for each data item (instance). The dictionary contains the key/value pairs:\n",
        "\n",
        "- `content`: The Cloud Storage path to the image.\n",
        "- `mime_type`: The content type. In our example, it is a `jpeg` file.\n",
        "\n",
        "For example:\n",
        "\n",
        "                        {'content': '[your-bucket]/file1.jpg', 'mime_type': 'jpeg'}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "make_batch_file:automl,image"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "\n",
        "import tensorflow as tf\n",
        "\n",
        "gcs_input_uri = BUCKET_NAME + \"/test.jsonl\"\n",
        "with tf.io.gfile.GFile(gcs_input_uri, \"w\") as f:\n",
        "    data = {\"content\": test_item_1, \"mime_type\": \"image/jpeg\"}\n",
        "    f.write(json.dumps(data) + \"\\n\")\n",
        "    data = {\"content\": test_item_2, \"mime_type\": \"image/jpeg\"}\n",
        "    f.write(json.dumps(data) + \"\\n\")\n",
        "\n",
        "print(gcs_input_uri)\n",
        "! gsutil cat $gcs_input_uri"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "run_automl_pipeline:icn"
      },
      "source": [
        "### Compile and execute the pipeline\n",
        "\n",
        "Next, you compile the pipeline and then exeute it. The pipeline takes the following parameters, which are passed as the dictionary `parameter_values`:\n",
        "\n",
        "- `import_file`: The Cloud Storage path to the dataset index file.\n",
        "- `batch_files`: A list of Cloud Storage paths to the input batch files.\n",
        "- `display_name`: The display name for the generated Vertex AI resources.\n",
        "- `project`: The project ID.\n",
        "- `region`: The region."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "run_automl_pipeline:icn"
      },
      "outputs": [],
      "source": [
        "compiler.Compiler().compile(\n",
        "    pipeline_func=pipeline, package_path=\"automl_icn_training.json\"\n",
        ")\n",
        "\n",
        "pipeline = aip.PipelineJob(\n",
        "    display_name=\"automl_icn_training\",\n",
        "    template_path=\"automl_icn_training.json\",\n",
        "    pipeline_root=PIPELINE_ROOT,\n",
        "    parameter_values={\n",
        "        \"import_file\": IMPORT_FILE,\n",
        "        \"batch_files\": [gcs_input_uri],\n",
        "        \"display_name\": \"flowers\" + TIMESTAMP,\n",
        "        \"project\": PROJECT_ID,\n",
        "        \"region\": REGION,\n",
        "    },\n",
        ")\n",
        "\n",
        "pipeline.run()\n",
        "\n",
        "! rm -f automl_icn_training.json"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view_pipeline_results:automl,icn"
      },
      "source": [
        "### View AutoML training pipeline results\n",
        "\n",
        "Finally, you will view the artifact outputs of each task in the pipeline."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "view_pipeline_results:automl,icn"
      },
      "outputs": [],
      "source": [
        "PROJECT_NUMBER = pipeline.gca_resource.name.split(\"/\")[1]\n",
        "print(PROJECT_NUMBER)\n",
        "\n",
        "\n",
        "def print_pipeline_output(job, output_task_name):\n",
        "    JOB_ID = job.name\n",
        "    print(JOB_ID)\n",
        "    for _ in range(len(job.gca_resource.job_detail.task_details)):\n",
        "        TASK_ID = job.gca_resource.job_detail.task_details[_].task_id\n",
        "        EXECUTE_OUTPUT = (\n",
        "            PIPELINE_ROOT\n",
        "            + \"/\"\n",
        "            + PROJECT_NUMBER\n",
        "            + \"/\"\n",
        "            + JOB_ID\n",
        "            + \"/\"\n",
        "            + output_task_name\n",
        "            + \"_\"\n",
        "            + str(TASK_ID)\n",
        "            + \"/executor_output.json\"\n",
        "        )\n",
        "        GCP_RESOURCES = (\n",
        "            PIPELINE_ROOT\n",
        "            + \"/\"\n",
        "            + PROJECT_NUMBER\n",
        "            + \"/\"\n",
        "            + JOB_ID\n",
        "            + \"/\"\n",
        "            + output_task_name\n",
        "            + \"_\"\n",
        "            + str(TASK_ID)\n",
        "            + \"/gcp_resources\"\n",
        "        )\n",
        "        if tf.io.gfile.exists(EXECUTE_OUTPUT):\n",
        "            ! gsutil cat $EXECUTE_OUTPUT\n",
        "            break\n",
        "        elif tf.io.gfile.exists(GCP_RESOURCES):\n",
        "            ! gsutil cat $GCP_RESOURCES\n",
        "            break\n",
        "\n",
        "    return EXECUTE_OUTPUT\n",
        "\n",
        "\n",
        "print(\"imagedataset-create\")\n",
        "artifacts = print_pipeline_output(pipeline, \"imagedataset-create\")\n",
        "print(\"\\n\\n\")\n",
        "print(\"automlimagetrainingjob-run\")\n",
        "artifacts = print_pipeline_output(pipeline, \"automlimagetrainingjob-run\")\n",
        "print(\"\\n\\n\")\n",
        "print(\"endpoint-create\")\n",
        "artifacts = print_pipeline_output(pipeline, \"endpoint-create\")\n",
        "print(\"\\n\\n\")\n",
        "print(\"model-deploy\")\n",
        "artifacts = print_pipeline_output(pipeline, \"model-deploy\")\n",
        "print(\"\\n\\n\")\n",
        "print(\"evaluateautomlmodelop\")\n",
        "artifacts = print_pipeline_output(pipeline, \"evaluateautomlmodelop\")\n",
        "print(\"\\n\\n\")\n",
        "print(\"model-batch-predict\")\n",
        "artifacts = print_pipeline_output(pipeline, \"model-batch-predict\")\n",
        "output = !gsutil cat $artifacts\n",
        "output = json.loads(output[0])\n",
        "print(\"\\n\\n\")\n",
        "print(\n",
        "    output[\"artifacts\"][\"batchpredictionjob\"][\"artifacts\"][0][\"metadata\"][\n",
        "        \"gcsOutputDirectory\"\n",
        "    ]\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "delete_pipeline"
      },
      "source": [
        "### Delete a pipeline job\n",
        "\n",
        "After a pipeline job is completed, you can delete the pipeline job with the method `delete()`.  Prior to completion, a pipeline job can be canceled with the method `cancel()`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "delete_pipeline"
      },
      "outputs": [],
      "source": [
        "pipeline.delete()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cleanup:mbsdk"
      },
      "source": [
        "# Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial:\n",
        "\n",
        "- Dataset\n",
        "- Pipeline\n",
        "- Model\n",
        "- Endpoint\n",
        "- AutoML Training Job\n",
        "- Batch Job\n",
        "- Custom Job\n",
        "- Hyperparameter Tuning Job\n",
        "- Cloud Storage Bucket"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cleanup:mbsdk"
      },
      "outputs": [],
      "source": [
        "delete_all = True\n",
        "\n",
        "if delete_all:\n",
        "    # Delete the dataset using the Vertex dataset object\n",
        "    try:\n",
        "        if \"dataset\" in globals():\n",
        "            dataset.delete()\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "    # Delete the model using the Vertex model object\n",
        "    try:\n",
        "        if \"model\" in globals():\n",
        "            model.delete()\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "    # Delete the endpoint using the Vertex endpoint object\n",
        "    try:\n",
        "        if \"endpoint\" in globals():\n",
        "            endpoint.undeploy_all()\n",
        "            endpoint.delete()\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "    # Delete the AutoML or Pipeline training job\n",
        "    try:\n",
        "        if \"dag\" in globals():\n",
        "            dag.delete()\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "    # Delete the custom training job\n",
        "    try:\n",
        "        if \"job\" in globals():\n",
        "            job.delete()\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "    # Delete the batch prediction job using the Vertex batch prediction object\n",
        "    try:\n",
        "        if \"batch_predict_job\" in globals():\n",
        "            batch_predict_job.delete()\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "    # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
        "    try:\n",
        "        if \"hpt_job\" in globals():\n",
        "            hpt_job.delete()\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "    if \"BUCKET_NAME\" in globals():\n",
        "        ! gsutil rm -r $BUCKET_NAME"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "get_started_with_automl_pipeline_components.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
<commit_msg>fix: breaking change in 0.3.0<commit_after>{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "copyright"
      },
      "outputs": [],
      "source": [
        "# Copyright 2021 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "title:generic,gcp"
      },
      "source": [
        "# E2E ML on GCP: MLOps stage 3 : formalization: get started with AutoML pipeline components\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td>\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_automl_pipeline_components.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
        "      View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://console.cloud.google.com/ai/platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage3/get_started_with_automl_pipeline_components.ipynb\">\n",
        "      Open in Google Cloud Notebooks\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "<br/><br/><br/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "overview:mlops"
      },
      "source": [
        "## Overview\n",
        "\n",
        "\n",
        "This tutorial demonstrates how to use Vertex AI for E2E MLOps on Google Cloud in production. This tutorial covers stage 3 : formalization: get started with AutoML pipeline components."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dataset:flowers,icn"
      },
      "source": [
        "### Dataset\n",
        "\n",
        "The dataset used for this tutorial is the [Flowers dataset](https://www.tensorflow.org/datasets/catalog/tf_flowers) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of flower an image is from a class of five flowers: daisy, dandelion, rose, sunflower, or tulip."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "objective:mlops,stage3,get_started_automl_pipeline_components"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you learn how to use prebuilt `Google Cloud Pipeline Components` for `Vertex AI AutoML`.\n",
        "\n",
        "This tutorial uses the following Google Cloud ML services:\n",
        "\n",
        "- `Vertex AI Pipelines`\n",
        "- `Vertex AI AutoML`\n",
        "- `Google Cloud Pipeline Components`\n",
        "- `Vertex AI Dataset, Model and Endpoint` resources\n",
        "- `Vertex AI Prediction`\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "- Construct a pipeline for:\n",
        "    - Training a Vertex AI AutoML trained model.\n",
        "    - Test the serving binary with a batch prediction job.\n",
        "    - Deploying a Vertex AI AutoML trained model.\n",
        "- Execute a Vertex AI pipeline."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "install_mlops"
      },
      "source": [
        "## Installations\n",
        "\n",
        "Install *one time* the packages for executing the MLOps notebooks."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "install_mlops"
      },
      "outputs": [],
      "source": [
        "ONCE_ONLY = False\n",
        "if ONCE_ONLY:\n",
        "    ! pip3 install -U tensorflow==2.5 $USER_FLAG\n",
        "    ! pip3 install -U tensorflow-data-validation==1.2 $USER_FLAG\n",
        "    ! pip3 install -U tensorflow-transform==1.2 $USER_FLAG\n",
        "    ! pip3 install -U tensorflow-io==0.18 $USER_FLAG\n",
        "    ! pip3 install --upgrade google-cloud-aiplatform[tensorboard] $USER_FLAG\n",
        "    ! pip3 install --upgrade google-cloud-pipeline-components $USER_FLAG\n",
        "    ! pip3 install --upgrade google-cloud-bigquery $USER_FLAG\n",
        "    ! pip3 install --upgrade google-cloud-logging $USER_FLAG\n",
        "    ! pip3 install --upgrade apache-beam[gcp] $USER_FLAG\n",
        "    ! pip3 install --upgrade pyarrow $USER_FLAG\n",
        "    ! pip3 install --upgrade cloudml-hypertune $USER_FLAG\n",
        "    ! pip3 install --upgrade kfp $USER_FLAG\n",
        "    ! pip3 install --upgrade torchvision $USER_FLAG\n",
        "    ! pip3 install --upgrade rpy2 $USER_FLAG"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "restart"
      },
      "source": [
        "### Restart the kernel\n",
        "\n",
        "Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "restart"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "if not os.getenv(\"IS_TESTING\"):\n",
        "    # Automatically restart kernel after installs\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "project_id"
      },
      "source": [
        "#### Set your project ID\n",
        "\n",
        "**If you don't know your project ID**, you may be able to get your project ID using `gcloud`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_project_id"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "autoset_project_id"
      },
      "outputs": [],
      "source": [
        "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
        "    # Get your GCP project id from gcloud\n",
        "    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
        "    PROJECT_ID = shell_output[0]\n",
        "    print(\"Project ID:\", PROJECT_ID)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_gcloud_project_id"
      },
      "outputs": [],
      "source": [
        "! gcloud config set project $PROJECT_ID"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "region"
      },
      "source": [
        "#### Region\n",
        "\n",
        "You can also change the `REGION` variable, which is used for operations\n",
        "throughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
        "\n",
        "- Americas: `us-central1`\n",
        "- Europe: `europe-west4`\n",
        "- Asia Pacific: `asia-east1`\n",
        "\n",
        "You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
        "\n",
        "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "region"
      },
      "outputs": [],
      "source": [
        "REGION = \"us-central1\"  # @param {type: \"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "timestamp"
      },
      "source": [
        "#### Timestamp\n",
        "\n",
        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "timestamp"
      },
      "outputs": [],
      "source": [
        "from datetime import datetime\n",
        "\n",
        "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bucket:mbsdk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "When you initialize the Vertex SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
        "\n",
        "Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bucket"
      },
      "outputs": [],
      "source": [
        "BUCKET_NAME = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "autoset_bucket"
      },
      "outputs": [],
      "source": [
        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
        "    BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_bucket"
      },
      "source": [
        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_bucket"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l $REGION $BUCKET_NAME"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "validate_bucket"
      },
      "source": [
        "Finally, validate access to your Cloud Storage bucket by examining its contents:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "validate_bucket"
      },
      "outputs": [],
      "source": [
        "! gsutil ls -al $BUCKET_NAME"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "set_service_account"
      },
      "source": [
        "#### Service Account\n",
        "\n",
        "**If you don't know your service account**, try to get your service account using `gcloud` command by executing the second cell below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_service_account"
      },
      "outputs": [],
      "source": [
        "SERVICE_ACCOUNT = \"[your-service-account]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "autoset_service_account"
      },
      "outputs": [],
      "source": [
        "if (\n",
        "    SERVICE_ACCOUNT == \"\"\n",
        "    or SERVICE_ACCOUNT is None\n",
        "    or SERVICE_ACCOUNT == \"[your-service-account]\"\n",
        "):\n",
        "    # Get your GCP project id from gcloud\n",
        "    shell_output = !gcloud auth list 2>/dev/null\n",
        "    SERVICE_ACCOUNT = shell_output[2].replace(\"*\", \"\").strip()\n",
        "    print(\"Service Account:\", SERVICE_ACCOUNT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "set_service_account:pipelines"
      },
      "source": [
        "#### Set service account access for Vertex AI Pipelines\n",
        "\n",
        "Run the following commands to grant your service account access to read and write pipeline artifacts in the bucket that you created in the previous step -- you only need to run these once per service account."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_service_account:pipelines"
      },
      "outputs": [],
      "source": [
        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_NAME\n",
        "\n",
        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_NAME"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "setup_vars"
      },
      "source": [
        "### Set up variables\n",
        "\n",
        "Next, set up some variables used throughout the tutorial.\n",
        "### Import libraries and define constants"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "import_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "import google.cloud.aiplatform as aip"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "import_tf"
      },
      "source": [
        "#### Import TensorFlow\n",
        "\n",
        "Import the TensorFlow package into your Python environment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "import_tf"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "import_kfp"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "\n",
        "from kfp import dsl\n",
        "from kfp.v2 import compiler\n",
        "from kfp.v2.dsl import component"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "init_aip:mbsdk,all"
      },
      "source": [
        "### Initialize Vertex AI SDK for Python\n",
        "\n",
        "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "init_aip:mbsdk,all"
      },
      "outputs": [],
      "source": [
        "aip.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_NAME)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "import_file:u_dataset,csv"
      },
      "source": [
        "#### Location of Cloud Storage training data.\n",
        "\n",
        "Now set the variable `IMPORT_FILE` to the location of the CSV index file in Cloud Storage."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "import_file:flowers,csv,icn"
      },
      "outputs": [],
      "source": [
        "IMPORT_FILE = (\n",
        "    \"gs://cloud-samples-data/vision/automl_classification/flowers/all_data_v2.csv\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_automl_eval_component"
      },
      "source": [
        "### Create AutoML model evaluation component\n",
        "\n",
        "The Vertex AI pre-built pipeline components does not currently have a component for retrieiving the model evaluations for a AutoML model. So, you will first write your own component, as follows:\n",
        "\n",
        "- Takes as input the region and Model artifacts returned from an AutoML training component.\n",
        "- Create a client interface to the Vertex AI Model service (`metadata[\"resource_name\"]).\n",
        "- Construct the resource ID for the model from the model artifact parameter.\n",
        "- Retrieve the model evaluation\n",
        "- Return the model evaluation as a string."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_automl_eval_component"
      },
      "outputs": [],
      "source": [
        "from kfp.v2.dsl import Artifact, Input, Model\n",
        "\n",
        "\n",
        "@component(packages_to_install=[\"google-cloud-aiplatform\"])\n",
        "def evaluateAutoMLModelOp(model: Input[Artifact], region: str) -> str:\n",
        "    import logging\n",
        "\n",
        "    import google.cloud.aiplatform.gapic as gapic\n",
        "\n",
        "    # Get a reference to the Model Service client\n",
        "    client_options = {\"api_endpoint\": f\"{region}-aiplatform.googleapis.com\"}\n",
        "    model_service_client = gapic.ModelServiceClient(client_options=client_options)\n",
        "\n",
        "    model_id = model.metadata[\"resourceName\"]\n",
        "\n",
        "    model_evaluations = model_service_client.list_model_evaluations(parent=model_id)\n",
        "    model_evaluation = list(model_evaluations)[0]\n",
        "    logging.info(model_evaluation)\n",
        "    return str(model_evaluation)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_automl_pipeline:icn"
      },
      "source": [
        "## Construct AutoML training pipeline\n",
        "\n",
        "In the example below, you construct a pipeline for training an AutoML model using pre-built Google Cloud Pipeline Components for AutoML, as follows:\n",
        "\n",
        "1. Use the prebuilt component `ImageDatasetCreateOp` to create a Vertex AI Dataset resource, where:\n",
        "    - The display name for the dataset is passed into the pipeline.\n",
        "    - The import file for the dataset is passed into the pipeline.\n",
        "    - The component returns the dataset resource as `outputs[\"dataset\"]`\n",
        "\n",
        "\n",
        "2. Use the prebuilt component `AutoMLImageTrainingJobRunOp` to train a Vertex AI AutoML Model resource, where:\n",
        "    - The display name for the dataset is passed into the pipeline.\n",
        "    - The dataset is the output from the `ImageDatasetCreateOp`.\n",
        "    - The component returns the model resource as `outputs[\"model\"]`.\n",
        "\n",
        "\n",
        "3. Use the prebuild component `ModelBatchPredictOp` to do a test batch prediction, where:\n",
        "    - The model is the output from the `AutoMLTrainingJobRunOp`.\n",
        "\n",
        "\n",
        "4. Use the prebuilt component `EndpointCreateOp` to create a Vertex AI Endpoint to deploy the trained model to, where:\n",
        "    - Since the component has no dependencies on other components, by default it would be executed in parallel with the model training.\n",
        "    - The `after(training_op)` is added to serialize its execution, so its only executed if the training operation completes successfully.\n",
        "    - The component returns the endpoint resource as `outputs[\"endpoint\"]`.\n",
        "\n",
        "\n",
        "5. Use the prebuilt component `ModelDeployOp` to deploy the trained AutoML model to, where:\n",
        "    - The display name for the dataset is passed into the pipeline.\n",
        "    - The model is the output from the `AutoMLTrainingJobRunOp`.\n",
        "    - The endpoint is the output from the `EndpointCreateOp`\n",
        "\n",
        "*Note:* Since each component is executed as a graph node in its own execution context, you pass the parameter `project` for each component op, in constrast to doing a `aip.init(project=project)` if this was a Python script calling the SDK methods directly within the same execution context."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_automl_pipeline:icn"
      },
      "outputs": [],
      "source": [
        "from google_cloud_pipeline_components import aiplatform as gcc_aip\n",
        "\n",
        "PIPELINE_ROOT = \"{}/pipeline_root/automl_icn_training\".format(BUCKET_NAME)\n",
        "DEPLOY_COMPUTE = \"n1-standard-4\"\n",
        "\n",
        "\n",
        "@dsl.pipeline(\n",
        "    name=\"automl-icn-training\", description=\"AutoML image classification training\"\n",
        ")\n",
        "def pipeline(\n",
        "    import_file: str,\n",
        "    batch_files: list,\n",
        "    display_name: str,\n",
        "    bucket: str = PIPELINE_ROOT,\n",
        "    project: str = PROJECT_ID,\n",
        "    region: str = REGION,\n",
        "):\n",
        "\n",
        "    dataset_op = gcc_aip.ImageDatasetCreateOp(\n",
        "        project=project,\n",
        "        display_name=display_name,\n",
        "        gcs_source=import_file,\n",
        "        import_schema_uri=aip.schema.dataset.ioformat.image.single_label_classification,\n",
        "    )\n",
        "\n",
        "    training_op = gcc_aip.AutoMLImageTrainingJobRunOp(\n",
        "        project=project,\n",
        "        display_name=display_name,\n",
        "        prediction_type=\"classification\",\n",
        "        model_type=\"CLOUD\",\n",
        "        dataset=dataset_op.outputs[\"dataset\"],\n",
        "        model_display_name=display_name,\n",
        "        training_fraction_split=0.6,\n",
        "        validation_fraction_split=0.2,\n",
        "        test_fraction_split=0.2,\n",
        "        budget_milli_node_hours=8000,\n",
        "    )\n",
        "\n",
        "    eval_op = evaluateAutoMLModelOp(model=training_op.outputs[\"model\"], region=region)\n",
        "\n",
        "    batch_op = gcc_aip.ModelBatchPredictOp(\n",
        "        project=project,\n",
        "        job_display_name=\"batch_predict_job\",\n",
        "        model=training_op.outputs[\"model\"],\n",
        "        gcs_source_uris=batch_files,\n",
        "        gcs_destination_output_uri_prefix=bucket,\n",
        "        instances_format=\"jsonl\",\n",
        "        predictions_format=\"jsonl\",\n",
        "        model_parameters={},\n",
        "        machine_type=DEPLOY_COMPUTE,\n",
        "        starting_replica_count=1,\n",
        "        max_replica_count=1,\n",
        "    ).after(eval_op)\n",
        "\n",
        "    endpoint_op = gcc_aip.EndpointCreateOp(\n",
        "        project=project,\n",
        "        location=region,\n",
        "        display_name=display_name,\n",
        "    ).after(batch_op)\n",
        "\n",
        "    deploy_op = gcc_aip.ModelDeployOp(\n",
        "        model=training_op.outputs[\"model\"],\n",
        "        endpoint=endpoint_op.outputs[\"endpoint\"],\n",
        "        automatic_resources_min_replica_count=1,\n",
        "        automatic_resources_max_replica_count=1,\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "get_test_items:batch_prediction"
      },
      "source": [
        "### Get test item(s)\n",
        "\n",
        "Now do a batch prediction to your Vertex model. You will use arbitrary examples out of the dataset as a test items. Don't be concerned that the examples were likely used in training the model -- we just want to demonstrate how to make a prediction."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "get_test_items:automl,icn,csv"
      },
      "outputs": [],
      "source": [
        "test_items = !gsutil cat $IMPORT_FILE | head -n2\n",
        "if len(str(test_items[0]).split(\",\")) == 3:\n",
        "    _, test_item_1, test_label_1 = str(test_items[0]).split(\",\")\n",
        "    _, test_item_2, test_label_2 = str(test_items[1]).split(\",\")\n",
        "else:\n",
        "    test_item_1, test_label_1 = str(test_items[0]).split(\",\")\n",
        "    test_item_2, test_label_2 = str(test_items[1]).split(\",\")\n",
        "\n",
        "print(test_item_1, test_label_1)\n",
        "print(test_item_2, test_label_2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "copy_test_items:batch_prediction"
      },
      "source": [
        "### Copy test item(s)\n",
        "\n",
        "For the batch prediction, copy the test items over to your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "copy_test_items:batch_prediction"
      },
      "outputs": [],
      "source": [
        "file_1 = test_item_1.split(\"/\")[-1]\n",
        "file_2 = test_item_2.split(\"/\")[-1]\n",
        "\n",
        "! gsutil cp $test_item_1 $BUCKET_NAME/$file_1\n",
        "! gsutil cp $test_item_2 $BUCKET_NAME/$file_2\n",
        "\n",
        "test_item_1 = BUCKET_NAME + \"/\" + file_1\n",
        "test_item_2 = BUCKET_NAME + \"/\" + file_2"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "make_batch_file:automl,image"
      },
      "source": [
        "### Make the batch input file\n",
        "\n",
        "Now make a batch input file, which you will store in your local Cloud Storage bucket. The batch input file can be either CSV or JSONL. You will use JSONL in this tutorial. For JSONL file, you make one dictionary entry per line for each data item (instance). The dictionary contains the key/value pairs:\n",
        "\n",
        "- `content`: The Cloud Storage path to the image.\n",
        "- `mime_type`: The content type. In our example, it is a `jpeg` file.\n",
        "\n",
        "For example:\n",
        "\n",
        "                        {'content': '[your-bucket]/file1.jpg', 'mime_type': 'jpeg'}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "make_batch_file:automl,image"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "\n",
        "import tensorflow as tf\n",
        "\n",
        "gcs_input_uri = BUCKET_NAME + \"/test.jsonl\"\n",
        "with tf.io.gfile.GFile(gcs_input_uri, \"w\") as f:\n",
        "    data = {\"content\": test_item_1, \"mime_type\": \"image/jpeg\"}\n",
        "    f.write(json.dumps(data) + \"\\n\")\n",
        "    data = {\"content\": test_item_2, \"mime_type\": \"image/jpeg\"}\n",
        "    f.write(json.dumps(data) + \"\\n\")\n",
        "\n",
        "print(gcs_input_uri)\n",
        "! gsutil cat $gcs_input_uri"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "run_automl_pipeline:icn"
      },
      "source": [
        "### Compile and execute the pipeline\n",
        "\n",
        "Next, you compile the pipeline and then exeute it. The pipeline takes the following parameters, which are passed as the dictionary `parameter_values`:\n",
        "\n",
        "- `import_file`: The Cloud Storage path to the dataset index file.\n",
        "- `batch_files`: A list of Cloud Storage paths to the input batch files.\n",
        "- `display_name`: The display name for the generated Vertex AI resources.\n",
        "- `project`: The project ID.\n",
        "- `region`: The region."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "run_automl_pipeline:icn"
      },
      "outputs": [],
      "source": [
        "compiler.Compiler().compile(\n",
        "    pipeline_func=pipeline, package_path=\"automl_icn_training.json\"\n",
        ")\n",
        "\n",
        "pipeline = aip.PipelineJob(\n",
        "    display_name=\"automl_icn_training\",\n",
        "    template_path=\"automl_icn_training.json\",\n",
        "    pipeline_root=PIPELINE_ROOT,\n",
        "    parameter_values={\n",
        "        \"import_file\": IMPORT_FILE,\n",
        "        \"batch_files\": [gcs_input_uri],\n",
        "        \"display_name\": \"flowers\" + TIMESTAMP,\n",
        "        \"project\": PROJECT_ID,\n",
        "        \"region\": REGION,\n",
        "    },\n",
        ")\n",
        "\n",
        "pipeline.run()\n",
        "\n",
        "! rm -f automl_icn_training.json"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view_pipeline_results:automl,icn"
      },
      "source": [
        "### View AutoML training pipeline results\n",
        "\n",
        "Finally, you will view the artifact outputs of each task in the pipeline."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "view_pipeline_results:automl,icn"
      },
      "outputs": [],
      "source": [
        "PROJECT_NUMBER = pipeline.gca_resource.name.split(\"/\")[1]\n",
        "print(PROJECT_NUMBER)\n",
        "\n",
        "\n",
        "def print_pipeline_output(job, output_task_name):\n",
        "    JOB_ID = job.name\n",
        "    print(JOB_ID)\n",
        "    for _ in range(len(job.gca_resource.job_detail.task_details)):\n",
        "        TASK_ID = job.gca_resource.job_detail.task_details[_].task_id\n",
        "        EXECUTE_OUTPUT = (\n",
        "            PIPELINE_ROOT\n",
        "            + \"/\"\n",
        "            + PROJECT_NUMBER\n",
        "            + \"/\"\n",
        "            + JOB_ID\n",
        "            + \"/\"\n",
        "            + output_task_name\n",
        "            + \"_\"\n",
        "            + str(TASK_ID)\n",
        "            + \"/executor_output.json\"\n",
        "        )\n",
        "        GCP_RESOURCES = (\n",
        "            PIPELINE_ROOT\n",
        "            + \"/\"\n",
        "            + PROJECT_NUMBER\n",
        "            + \"/\"\n",
        "            + JOB_ID\n",
        "            + \"/\"\n",
        "            + output_task_name\n",
        "            + \"_\"\n",
        "            + str(TASK_ID)\n",
        "            + \"/gcp_resources\"\n",
        "        )\n",
        "        if tf.io.gfile.exists(EXECUTE_OUTPUT):\n",
        "            ! gsutil cat $EXECUTE_OUTPUT\n",
        "            break\n",
        "        elif tf.io.gfile.exists(GCP_RESOURCES):\n",
        "            ! gsutil cat $GCP_RESOURCES\n",
        "            break\n",
        "\n",
        "    return EXECUTE_OUTPUT\n",
        "\n",
        "\n",
        "print(\"imagedataset-create\")\n",
        "artifacts = print_pipeline_output(pipeline, \"imagedataset-create\")\n",
        "print(\"\\n\\n\")\n",
        "print(\"automlimagetrainingjob-run\")\n",
        "artifacts = print_pipeline_output(pipeline, \"automlimagetrainingjob-run\")\n",
        "print(\"\\n\\n\")\n",
        "print(\"endpoint-create\")\n",
        "artifacts = print_pipeline_output(pipeline, \"endpoint-create\")\n",
        "print(\"\\n\\n\")\n",
        "print(\"model-deploy\")\n",
        "artifacts = print_pipeline_output(pipeline, \"model-deploy\")\n",
        "print(\"\\n\\n\")\n",
        "print(\"evaluateautomlmodelop\")\n",
        "artifacts = print_pipeline_output(pipeline, \"evaluateautomlmodelop\")\n",
        "print(\"\\n\\n\")\n",
        "print(\"model-batch-predict\")\n",
        "artifacts = print_pipeline_output(pipeline, \"model-batch-predict\")\n",
        "output = !gsutil cat $artifacts\n",
        "output = json.loads(output[0])\n",
        "print(\"\\n\\n\")\n",
        "print(\n",
        "    output[\"artifacts\"][\"batchpredictionjob\"][\"artifacts\"][0][\"metadata\"][\n",
        "        \"gcsOutputDirectory\"\n",
        "    ]\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "delete_pipeline"
      },
      "source": [
        "### Delete a pipeline job\n",
        "\n",
        "After a pipeline job is completed, you can delete the pipeline job with the method `delete()`.  Prior to completion, a pipeline job can be canceled with the method `cancel()`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "delete_pipeline"
      },
      "outputs": [],
      "source": [
        "pipeline.delete()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cleanup:mbsdk"
      },
      "source": [
        "# Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial:\n",
        "\n",
        "- Dataset\n",
        "- Pipeline\n",
        "- Model\n",
        "- Endpoint\n",
        "- AutoML Training Job\n",
        "- Batch Job\n",
        "- Custom Job\n",
        "- Hyperparameter Tuning Job\n",
        "- Cloud Storage Bucket"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cleanup:mbsdk"
      },
      "outputs": [],
      "source": [
        "delete_all = True\n",
        "\n",
        "if delete_all:\n",
        "    # Delete the dataset using the Vertex dataset object\n",
        "    try:\n",
        "        if \"dataset\" in globals():\n",
        "            dataset.delete()\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "    # Delete the model using the Vertex model object\n",
        "    try:\n",
        "        if \"model\" in globals():\n",
        "            model.delete()\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "    # Delete the endpoint using the Vertex endpoint object\n",
        "    try:\n",
        "        if \"endpoint\" in globals():\n",
        "            endpoint.undeploy_all()\n",
        "            endpoint.delete()\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "    # Delete the AutoML or Pipeline training job\n",
        "    try:\n",
        "        if \"dag\" in globals():\n",
        "            dag.delete()\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "    # Delete the custom training job\n",
        "    try:\n",
        "        if \"job\" in globals():\n",
        "            job.delete()\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "    # Delete the batch prediction job using the Vertex batch prediction object\n",
        "    try:\n",
        "        if \"batch_predict_job\" in globals():\n",
        "            batch_predict_job.delete()\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "    # Delete the hyperparameter tuning job using the Vertex hyperparameter tuning object\n",
        "    try:\n",
        "        if \"hpt_job\" in globals():\n",
        "            hpt_job.delete()\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "\n",
        "    if \"BUCKET_NAME\" in globals():\n",
        "        ! gsutil rm -r $BUCKET_NAME"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "get_started_with_automl_pipeline_components.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}

{'fraction_non_alphanumeric': 0.15974253150025966, 'fraction_numerical': 0.003266187443627518, 'mean_word_length': 1.8132328630194918, 'pattern_counts': {'":': 608, '<': 31, '<?xml version=': 0, '>': 37, 'https://': 18, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5455435', 'n_tokens_mistral': 23155, 'n_tokens_neox': 22889, 'n_words': 5291}