<commit_before>{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "copyright"
      },
      "outputs": [],
      "source": [
        "# Copyright 2021 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "title:generic"
      },
      "source": [
        "# Vertex AI Pipelines: AutoML image classification pipelines using google-cloud-pipeline-components\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td>\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/official/pipelines/google_cloud_pipeline_components_automl_images.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/official/pipelines/google_cloud_pipeline_components_automl_images.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
        "      View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/official/pipelines/google_cloud_pipeline_components_automl_images.ipynb\">\n",
        "      Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "<br/><br/><br/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "overview:pipelines,automl"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This notebook shows how to use the components defined in [`google_cloud_pipeline_components`](https://github.com/kubeflow/pipelines/tree/master/components/google-cloud) to build an AutoML image classification workflow on [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dataset:flowers,icn"
      },
      "source": [
        "### Dataset\n",
        "\n",
        "The dataset used for this tutorial is the [Flowers dataset](https://www.tensorflow.org/datasets/catalog/tf_flowers) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of flower an image is from a class of five flowers: daisy, dandelion, rose, sunflower, or tulip."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "objective:pipelines,automl"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you create an AutoML image classification using a pipeline with components from  `google_cloud_pipeline_components`.\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "- Create a `Dataset` resource.\n",
        "- Train an AutoML `Model` resource.\n",
        "- Creates an `Endpoint` resource.\n",
        "- Deploys the `Model` resource to the `Endpoint` resource.\n",
        "\n",
        "The components are [documented here](https://google-cloud-pipeline-components.readthedocs.io/en/latest/google_cloud_pipeline_components.aiplatform.html#module-google_cloud_pipeline_components.aiplatform)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "costs"
      },
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI\n",
        "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
        "Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "setup_local"
      },
      "source": [
        "### Set up your local development environment\n",
        "\n",
        "If you are using Colab or Google Cloud Notebook, your environment already meets all the requirements to run this notebook. You can skip this step.\n",
        "\n",
        "Otherwise, make sure your environment meets this notebook's requirements. You need the following:\n",
        "\n",
        "- The Cloud Storage SDK\n",
        "- Git\n",
        "- Python 3\n",
        "- virtualenv\n",
        "- Jupyter notebook running in a virtual environment with Python 3\n",
        "\n",
        "The Cloud Storage guide to [Setting up a Python development environment](https://cloud.google.com/python/setup) and the [Jupyter installation guide](https://jupyter.org/install) provide detailed instructions for meeting these requirements. The following steps provide a condensed set of instructions:\n",
        "\n",
        "1. [Install and initialize the SDK](https://cloud.google.com/sdk/docs/).\n",
        "\n",
        "2. [Install Python 3](https://cloud.google.com/python/setup#installing_python).\n",
        "\n",
        "3. [Install virtualenv](Ihttps://cloud.google.com/python/setup#installing_and_using_virtualenv) and create a virtual environment that uses Python 3.\n",
        "\n",
        "4. Activate that environment and run `pip3 install Jupyter` in a terminal shell to install Jupyter.\n",
        "\n",
        "5. Run `jupyter notebook` on the command line in a terminal shell to launch Jupyter.\n",
        "\n",
        "6. Open this notebook in the Jupyter Notebook Dashboard.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "install_aip:mbsdk"
      },
      "source": [
        "## Installation\n",
        "\n",
        "Install the latest version of Vertex AI SDK for Python."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "install_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "# Google Cloud Notebook\n",
        "if os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
        "    USER_FLAG = \"--user\"\n",
        "else:\n",
        "    USER_FLAG = \"\"\n",
        "\n",
        "! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "install_storage"
      },
      "source": [
        "Install the latest GA version of *google-cloud-storage* library as well."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "install_storage"
      },
      "outputs": [],
      "source": [
        "! pip3 install -U google-cloud-storage $USER_FLAG"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "install_gcpc"
      },
      "source": [
        "Install the latest GA version of *google-cloud-pipeline-components* library as well."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "install_gcpc"
      },
      "outputs": [],
      "source": [
        "! pip3 install $USER kfp google-cloud-pipeline-components --upgrade"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "restart"
      },
      "source": [
        "### Restart the kernel\n",
        "\n",
        "Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "restart"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "if not os.getenv(\"IS_TESTING\"):\n",
        "    # Automatically restart kernel after installs\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "check_versions"
      },
      "source": [
        "Check the versions of the packages you installed.  The KFP SDK version should be >=1.6."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "check_versions:kfp,gcpc"
      },
      "outputs": [],
      "source": [
        "! python3 -c \"import kfp; print('KFP SDK version: {}'.format(kfp.__version__))\"\n",
        "! python3 -c \"import google_cloud_pipeline_components; print('google_cloud_pipeline_components version: {}'.format(google_cloud_pipeline_components.__version__))\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "before_you_begin:nogpu"
      },
      "source": [
        "## Before you begin\n",
        "\n",
        "### GPU runtime\n",
        "\n",
        "This tutorial does not require a GPU runtime.\n",
        "\n",
        "### Set up your Google Cloud project\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
        "\n",
        "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
        "\n",
        "3. [Enable the Vertex AI APIs, Compute Engine APIs, and Cloud Storage.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component,storage-component.googleapis.com)\n",
        "\n",
        "4. [The Google Cloud SDK](https://cloud.google.com/sdk) is already installed in Google Cloud Notebook.\n",
        "\n",
        "5. Enter your project ID in the cell below. Then run the  cell to make sure the\n",
        "Cloud SDK uses the right project for all the commands in this notebook.\n",
        "\n",
        "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_project_id"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "autoset_project_id"
      },
      "outputs": [],
      "source": [
        "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
        "    # Get your GCP project id from gcloud\n",
        "    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
        "    PROJECT_ID = shell_output[0]\n",
        "    print(\"Project ID:\", PROJECT_ID)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_gcloud_project_id"
      },
      "outputs": [],
      "source": [
        "! gcloud config set project $PROJECT_ID"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "region"
      },
      "source": [
        "#### Region\n",
        "\n",
        "You can also change the `REGION` variable, which is used for operations\n",
        "throughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
        "\n",
        "- Americas: `us-central1`\n",
        "- Europe: `europe-west4`\n",
        "- Asia Pacific: `asia-east1`\n",
        "\n",
        "You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
        "\n",
        "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "region"
      },
      "outputs": [],
      "source": [
        "REGION = \"us-central1\"  # @param {type: \"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "timestamp"
      },
      "source": [
        "#### Timestamp\n",
        "\n",
        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "timestamp"
      },
      "outputs": [],
      "source": [
        "from datetime import datetime\n",
        "\n",
        "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gcp_authenticate"
      },
      "source": [
        "### Authenticate your Google Cloud account\n",
        "\n",
        "**If you are using Google Cloud Notebook**, your environment is already authenticated. Skip this step.\n",
        "\n",
        "**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.\n",
        "\n",
        "**Otherwise**, follow these steps:\n",
        "\n",
        "In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.\n",
        "\n",
        "**Click Create service account**.\n",
        "\n",
        "In the **Service account name** field, enter a name, and click **Create**.\n",
        "\n",
        "In the **Grant this service account access to project** section, click the Role drop-down list. Type \"Vertex\" into the filter box, and select **Vertex Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
        "\n",
        "Click Create. A JSON file that contains your key downloads to your local environment.\n",
        "\n",
        "Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gcp_authenticate"
      },
      "outputs": [],
      "source": [
        "# If you are running this notebook in Colab, run this cell and follow the\n",
        "# instructions to authenticate your GCP account. This provides access to your\n",
        "# Cloud Storage bucket and lets you submit training jobs and prediction\n",
        "# requests.\n",
        "\n",
        "import os\n",
        "import sys\n",
        "\n",
        "# If on Google Cloud Notebook, then don't execute this code\n",
        "if not os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
        "    if \"google.colab\" in sys.modules:\n",
        "        from google.colab import auth as google_auth\n",
        "\n",
        "        google_auth.authenticate_user()\n",
        "\n",
        "    # If you are running this notebook locally, replace the string below with the\n",
        "    # path to your service account key and run this cell to authenticate your GCP\n",
        "    # account.\n",
        "    elif not os.getenv(\"IS_TESTING\"):\n",
        "        %env GOOGLE_APPLICATION_CREDENTIALS ''"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bucket:mbsdk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "When you initialize the Vertex AI SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
        "\n",
        "Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bucket"
      },
      "outputs": [],
      "source": [
        "BUCKET_NAME = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "autoset_bucket"
      },
      "outputs": [],
      "source": [
        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
        "    BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_bucket"
      },
      "source": [
        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_bucket"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l $REGION $BUCKET_NAME"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "validate_bucket"
      },
      "source": [
        "Finally, validate access to your Cloud Storage bucket by examining its contents:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "validate_bucket"
      },
      "outputs": [],
      "source": [
        "! gsutil ls -al $BUCKET_NAME"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "set_service_account"
      },
      "source": [
        "#### Service Account\n",
        "\n",
        "**If you don't know your service account**, try to get your service account using `gcloud` command by executing the second cell below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_service_account"
      },
      "outputs": [],
      "source": [
        "SERVICE_ACCOUNT = \"[your-service-account]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "autoset_service_account"
      },
      "outputs": [],
      "source": [
        "if (\n",
        "    SERVICE_ACCOUNT == \"\"\n",
        "    or SERVICE_ACCOUNT is None\n",
        "    or SERVICE_ACCOUNT == \"[your-service-account]\"\n",
        "):\n",
        "    # Get your GCP project id from gcloud\n",
        "    shell_output = !gcloud auth list 2>/dev/null\n",
        "    SERVICE_ACCOUNT = shell_output[2].strip()\n",
        "    print(\"Service Account:\", SERVICE_ACCOUNT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "set_service_account:pipelines"
      },
      "source": [
        "#### Set service account access for Vertex AI Pipelines\n",
        "\n",
        "Run the following commands to grant your service account access to read and write pipeline artifacts in the bucket that you created in the previous step -- you only need to run these once per service account."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_service_account:pipelines"
      },
      "outputs": [],
      "source": [
        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_NAME\n",
        "\n",
        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_NAME"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "setup_vars"
      },
      "source": [
        "### Set up variables\n",
        "\n",
        "Next, set up some variables used throughout the tutorial.\n",
        "### Import libraries and define constants"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "import_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "import google.cloud.aiplatform as aip"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pipeline_constants"
      },
      "source": [
        "#### Vertex AI Pipelines constants\n",
        "\n",
        "Setup up the following constants for Vertex AI Pipelines:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pipeline_constants"
      },
      "outputs": [],
      "source": [
        "PIPELINE_ROOT = \"{}/pipeline_root/flowers\".format(BUCKET_NAME)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "additional_imports"
      },
      "source": [
        "Additional imports."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "import_pipelines:min"
      },
      "outputs": [],
      "source": [
        "import kfp"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "init_aip:mbsdk"
      },
      "source": [
        "## Initialize Vertex AI SDK for Python\n",
        "\n",
        "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "init_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "aip.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "define_pipeline:gcpc,automl,flowers,icn"
      },
      "source": [
        "## Define AutoML image classification model pipeline that uses components from `google_cloud_pipeline_components`\n",
        "\n",
        "Next, you define the pipeline.\n",
        "\n",
        "Create and deploy an AutoML image classification `Model` resource using a `Dataset` resource."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "define_pipeline:gcpc,automl,flowers,icn"
      },
      "outputs": [],
      "source": [
        "@kfp.dsl.pipeline(name=\"automl-image-training-v2\")\n",
        "def pipeline(project: str = PROJECT_ID, region: str = REGION):\n",
        "    from google_cloud_pipeline_components import aiplatform as gcc_aip\n",
        "    from google_cloud_pipeline_components.v1.endpoint import (EndpointCreateOp,\n",
        "                                                              ModelDeployOp)\n",
        "\n",
        "    ds_op = gcc_aip.ImageDatasetCreateOp(\n",
        "        project=project,\n",
        "        display_name=\"flowers\",\n",
        "        gcs_source=\"gs://cloud-samples-data/vision/automl_classification/flowers/all_data_v2.csv\",\n",
        "        import_schema_uri=aip.schema.dataset.ioformat.image.single_label_classification,\n",
        "    )\n",
        "\n",
        "    training_job_run_op = gcc_aip.AutoMLImageTrainingJobRunOp(\n",
        "        project=project,\n",
        "        display_name=\"train-automl-flowers\",\n",
        "        prediction_type=\"classification\",\n",
        "        model_type=\"CLOUD\",\n",
        "        dataset=ds_op.outputs[\"dataset\"],\n",
        "        model_display_name=\"train-automl-flowers\",\n",
        "        training_fraction_split=0.6,\n",
        "        validation_fraction_split=0.2,\n",
        "        test_fraction_split=0.2,\n",
        "        budget_milli_node_hours=8000,\n",
        "    )\n",
        "\n",
        "    endpoint_op = EndpointCreateOp(\n",
        "        project=project,\n",
        "        location=region,\n",
        "        display_name=\"train-automl-flowers\",\n",
        "    )\n",
        "\n",
        "    ModelDeployOp(\n",
        "        model=training_job_run_op.outputs[\"model\"],\n",
        "        endpoint=endpoint_op.outputs[\"endpoint\"],\n",
        "        automatic_resources_min_replica_count=1,\n",
        "        automatic_resources_max_replica_count=1,\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "compile_pipeline"
      },
      "source": [
        "## Compile the pipeline\n",
        "\n",
        "Next, compile the pipeline."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "compile_pipeline"
      },
      "outputs": [],
      "source": [
        "from kfp.v2 import compiler  # noqa: F811\n",
        "\n",
        "compiler.Compiler().compile(\n",
        "    pipeline_func=pipeline,\n",
        "    package_path=\"image classification_pipeline.json\".replace(\" \", \"_\"),\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "run_pipeline:automl,image"
      },
      "source": [
        "## Run the pipeline\n",
        "\n",
        "Next, run the pipeline."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "run_pipeline:automl,image"
      },
      "outputs": [],
      "source": [
        "DISPLAY_NAME = \"flowers_\" + TIMESTAMP\n",
        "\n",
        "job = aip.PipelineJob(\n",
        "    display_name=DISPLAY_NAME,\n",
        "    template_path=\"image classification_pipeline.json\".replace(\" \", \"_\"),\n",
        "    pipeline_root=PIPELINE_ROOT,\n",
        "    enable_caching=False,\n",
        ")\n",
        "\n",
        "job.run()\n",
        "\n",
        "! rm image_classification_pipeline.json"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view_pipeline_run:automl,image"
      },
      "source": [
        "Click on the generated link to see your run in the Cloud Console.\n",
        "\n",
        "<!-- It should look something like this as it is running:\n",
        "\n",
        "<a href=\"https://storage.googleapis.com/amy-jo/images/mp/automl_tabular_classif.png\" target=\"_blank\"><img src=\"https://storage.googleapis.com/amy-jo/images/mp/automl_tabular_classif.png\" width=\"40%\"/></a> -->\n",
        "\n",
        "In the UI, many of the pipeline DAG nodes will expand or collapse when you click on them. Here is a partially-expanded view of the DAG (click image to see larger version).\n",
        "\n",
        "<a href=\"https://storage.googleapis.com/amy-jo/images/mp/automl_image_classif.png\" target=\"_blank\"><img src=\"https://storage.googleapis.com/amy-jo/images/mp/automl_image_classif.png\" width=\"40%\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cleanup:pipelines"
      },
      "source": [
        "# Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial -- *Note:* this is auto-generated and not all resources may be applicable for this tutorial:\n",
        "\n",
        "- Dataset\n",
        "- Pipeline\n",
        "- Model\n",
        "- Endpoint\n",
        "- Batch Job\n",
        "- Custom Job\n",
        "- Hyperparameter Tuning Job\n",
        "- Cloud Storage Bucket"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cleanup:pipelines"
      },
      "outputs": [],
      "source": [
        "delete_dataset = True\n",
        "delete_pipeline = True\n",
        "delete_model = True\n",
        "delete_endpoint = True\n",
        "delete_batchjob = True\n",
        "delete_customjob = True\n",
        "delete_hptjob = True\n",
        "delete_bucket = True\n",
        "\n",
        "try:\n",
        "    if delete_model and \"DISPLAY_NAME\" in globals():\n",
        "        models = aip.Model.list(\n",
        "            filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
        "        )\n",
        "        model = models[0]\n",
        "        aip.Model.delete(model)\n",
        "        print(\"Deleted model:\", model)\n",
        "except Exception as e:\n",
        "    print(e)\n",
        "\n",
        "try:\n",
        "    if delete_endpoint and \"DISPLAY_NAME\" in globals():\n",
        "        endpoints = aip.Endpoint.list(\n",
        "            filter=f\"display_name={DISPLAY_NAME}_endpoint\", order_by=\"create_time\"\n",
        "        )\n",
        "        endpoint = endpoints[0]\n",
        "        endpoint.undeploy_all()\n",
        "        aip.Endpoint.delete(endpoint.resource_name)\n",
        "        print(\"Deleted endpoint:\", endpoint)\n",
        "except Exception as e:\n",
        "    print(e)\n",
        "\n",
        "if delete_dataset and \"DISPLAY_NAME\" in globals():\n",
        "    if \"image\" == \"tabular\":\n",
        "        try:\n",
        "            datasets = aip.TabularDataset.list(\n",
        "                filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
        "            )\n",
        "            dataset = datasets[0]\n",
        "            aip.TabularDataset.delete(dataset.resource_name)\n",
        "            print(\"Deleted dataset:\", dataset)\n",
        "        except Exception as e:\n",
        "            print(e)\n",
        "\n",
        "    if \"image\" == \"image\":\n",
        "        try:\n",
        "            datasets = aip.ImageDataset.list(\n",
        "                filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
        "            )\n",
        "            dataset = datasets[0]\n",
        "            aip.ImageDataset.delete(dataset.resource_name)\n",
        "            print(\"Deleted dataset:\", dataset)\n",
        "        except Exception as e:\n",
        "            print(e)\n",
        "\n",
        "    if \"image\" == \"text\":\n",
        "        try:\n",
        "            datasets = aip.TextDataset.list(\n",
        "                filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
        "            )\n",
        "            dataset = datasets[0]\n",
        "            aip.TextDataset.delete(dataset.resource_name)\n",
        "            print(\"Deleted dataset:\", dataset)\n",
        "        except Exception as e:\n",
        "            print(e)\n",
        "\n",
        "    if \"image\" == \"video\":\n",
        "        try:\n",
        "            datasets = aip.VideoDataset.list(\n",
        "                filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
        "            )\n",
        "            dataset = datasets[0]\n",
        "            aip.VideoDataset.delete(dataset.resource_name)\n",
        "            print(\"Deleted dataset:\", dataset)\n",
        "        except Exception as e:\n",
        "            print(e)\n",
        "\n",
        "try:\n",
        "    if delete_pipeline and \"DISPLAY_NAME\" in globals():\n",
        "        pipelines = aip.PipelineJob.list(\n",
        "            filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
        "        )\n",
        "        pipeline = pipelines[0]\n",
        "        aip.PipelineJob.delete(pipeline.resource_name)\n",
        "        print(\"Deleted pipeline:\", pipeline)\n",
        "except Exception as e:\n",
        "    print(e)\n",
        "\n",
        "if delete_bucket and \"BUCKET_NAME\" in globals():\n",
        "    ! gsutil rm -r $BUCKET_NAME"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "google_cloud_pipeline_components_automl_images.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
<commit_msg>fix: service account<commit_after>{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "copyright"
      },
      "outputs": [],
      "source": [
        "# Copyright 2021 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "title:generic"
      },
      "source": [
        "# Vertex AI Pipelines: AutoML image classification pipelines using google-cloud-pipeline-components\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td>\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/official/pipelines/google_cloud_pipeline_components_automl_images.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/official/pipelines/google_cloud_pipeline_components_automl_images.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
        "      View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/official/pipelines/google_cloud_pipeline_components_automl_images.ipynb\">\n",
        "      Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "<br/><br/><br/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "overview:pipelines,automl"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This notebook shows how to use the components defined in [`google_cloud_pipeline_components`](https://github.com/kubeflow/pipelines/tree/master/components/google-cloud) to build an AutoML image classification workflow on [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dataset:flowers,icn"
      },
      "source": [
        "### Dataset\n",
        "\n",
        "The dataset used for this tutorial is the [Flowers dataset](https://www.tensorflow.org/datasets/catalog/tf_flowers) from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/overview). The version of the dataset you will use in this tutorial is stored in a public Cloud Storage bucket. The trained model predicts the type of flower an image is from a class of five flowers: daisy, dandelion, rose, sunflower, or tulip."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "objective:pipelines,automl"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you create an AutoML image classification using a pipeline with components from  `google_cloud_pipeline_components`.\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "- Create a `Dataset` resource.\n",
        "- Train an AutoML `Model` resource.\n",
        "- Creates an `Endpoint` resource.\n",
        "- Deploys the `Model` resource to the `Endpoint` resource.\n",
        "\n",
        "The components are [documented here](https://google-cloud-pipeline-components.readthedocs.io/en/latest/google_cloud_pipeline_components.aiplatform.html#module-google_cloud_pipeline_components.aiplatform)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "costs"
      },
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI\n",
        "pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage\n",
        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
        "Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "setup_local"
      },
      "source": [
        "### Set up your local development environment\n",
        "\n",
        "If you are using Colab or Google Cloud Notebook, your environment already meets all the requirements to run this notebook. You can skip this step.\n",
        "\n",
        "Otherwise, make sure your environment meets this notebook's requirements. You need the following:\n",
        "\n",
        "- The Cloud Storage SDK\n",
        "- Git\n",
        "- Python 3\n",
        "- virtualenv\n",
        "- Jupyter notebook running in a virtual environment with Python 3\n",
        "\n",
        "The Cloud Storage guide to [Setting up a Python development environment](https://cloud.google.com/python/setup) and the [Jupyter installation guide](https://jupyter.org/install) provide detailed instructions for meeting these requirements. The following steps provide a condensed set of instructions:\n",
        "\n",
        "1. [Install and initialize the SDK](https://cloud.google.com/sdk/docs/).\n",
        "\n",
        "2. [Install Python 3](https://cloud.google.com/python/setup#installing_python).\n",
        "\n",
        "3. [Install virtualenv](Ihttps://cloud.google.com/python/setup#installing_and_using_virtualenv) and create a virtual environment that uses Python 3.\n",
        "\n",
        "4. Activate that environment and run `pip3 install Jupyter` in a terminal shell to install Jupyter.\n",
        "\n",
        "5. Run `jupyter notebook` on the command line in a terminal shell to launch Jupyter.\n",
        "\n",
        "6. Open this notebook in the Jupyter Notebook Dashboard.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "install_aip:mbsdk"
      },
      "source": [
        "## Installation\n",
        "\n",
        "Install the latest version of Vertex AI SDK for Python."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "install_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "# Google Cloud Notebook\n",
        "if os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
        "    USER_FLAG = \"--user\"\n",
        "else:\n",
        "    USER_FLAG = \"\"\n",
        "\n",
        "! pip3 install --upgrade google-cloud-aiplatform $USER_FLAG"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "install_storage"
      },
      "source": [
        "Install the latest GA version of *google-cloud-storage* library as well."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "install_storage"
      },
      "outputs": [],
      "source": [
        "! pip3 install -U google-cloud-storage $USER_FLAG"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "install_gcpc"
      },
      "source": [
        "Install the latest GA version of *google-cloud-pipeline-components* library as well."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "install_gcpc"
      },
      "outputs": [],
      "source": [
        "! pip3 install $USER kfp google-cloud-pipeline-components --upgrade"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "restart"
      },
      "source": [
        "### Restart the kernel\n",
        "\n",
        "Once you've installed the additional packages, you need to restart the notebook kernel so it can find the packages."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "restart"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "if not os.getenv(\"IS_TESTING\"):\n",
        "    # Automatically restart kernel after installs\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "check_versions"
      },
      "source": [
        "Check the versions of the packages you installed.  The KFP SDK version should be >=1.6."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "check_versions:kfp,gcpc"
      },
      "outputs": [],
      "source": [
        "! python3 -c \"import kfp; print('KFP SDK version: {}'.format(kfp.__version__))\"\n",
        "! python3 -c \"import google_cloud_pipeline_components; print('google_cloud_pipeline_components version: {}'.format(google_cloud_pipeline_components.__version__))\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "before_you_begin:nogpu"
      },
      "source": [
        "## Before you begin\n",
        "\n",
        "### GPU runtime\n",
        "\n",
        "This tutorial does not require a GPU runtime.\n",
        "\n",
        "### Set up your Google Cloud project\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
        "\n",
        "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
        "\n",
        "3. [Enable the Vertex AI APIs, Compute Engine APIs, and Cloud Storage.](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component,storage-component.googleapis.com)\n",
        "\n",
        "4. [The Google Cloud SDK](https://cloud.google.com/sdk) is already installed in Google Cloud Notebook.\n",
        "\n",
        "5. Enter your project ID in the cell below. Then run the  cell to make sure the\n",
        "Cloud SDK uses the right project for all the commands in this notebook.\n",
        "\n",
        "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_project_id"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "autoset_project_id"
      },
      "outputs": [],
      "source": [
        "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
        "    # Get your GCP project id from gcloud\n",
        "    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
        "    PROJECT_ID = shell_output[0]\n",
        "    print(\"Project ID:\", PROJECT_ID)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_gcloud_project_id"
      },
      "outputs": [],
      "source": [
        "! gcloud config set project $PROJECT_ID"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "region"
      },
      "source": [
        "#### Region\n",
        "\n",
        "You can also change the `REGION` variable, which is used for operations\n",
        "throughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.\n",
        "\n",
        "- Americas: `us-central1`\n",
        "- Europe: `europe-west4`\n",
        "- Asia Pacific: `asia-east1`\n",
        "\n",
        "You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.\n",
        "\n",
        "Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "region"
      },
      "outputs": [],
      "source": [
        "REGION = \"us-central1\"  # @param {type: \"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "timestamp"
      },
      "source": [
        "#### Timestamp\n",
        "\n",
        "If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append the timestamp onto the name of resources you create in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "timestamp"
      },
      "outputs": [],
      "source": [
        "from datetime import datetime\n",
        "\n",
        "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gcp_authenticate"
      },
      "source": [
        "### Authenticate your Google Cloud account\n",
        "\n",
        "**If you are using Google Cloud Notebook**, your environment is already authenticated. Skip this step.\n",
        "\n",
        "**If you are using Colab**, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.\n",
        "\n",
        "**Otherwise**, follow these steps:\n",
        "\n",
        "In the Cloud Console, go to the [Create service account key](https://console.cloud.google.com/apis/credentials/serviceaccountkey) page.\n",
        "\n",
        "**Click Create service account**.\n",
        "\n",
        "In the **Service account name** field, enter a name, and click **Create**.\n",
        "\n",
        "In the **Grant this service account access to project** section, click the Role drop-down list. Type \"Vertex\" into the filter box, and select **Vertex Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
        "\n",
        "Click Create. A JSON file that contains your key downloads to your local environment.\n",
        "\n",
        "Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gcp_authenticate"
      },
      "outputs": [],
      "source": [
        "# If you are running this notebook in Colab, run this cell and follow the\n",
        "# instructions to authenticate your GCP account. This provides access to your\n",
        "# Cloud Storage bucket and lets you submit training jobs and prediction\n",
        "# requests.\n",
        "\n",
        "import os\n",
        "import sys\n",
        "\n",
        "# If on Google Cloud Notebook, then don't execute this code\n",
        "if not os.path.exists(\"/opt/deeplearning/metadata/env_version\"):\n",
        "    if \"google.colab\" in sys.modules:\n",
        "        from google.colab import auth as google_auth\n",
        "\n",
        "        google_auth.authenticate_user()\n",
        "\n",
        "    # If you are running this notebook locally, replace the string below with the\n",
        "    # path to your service account key and run this cell to authenticate your GCP\n",
        "    # account.\n",
        "    elif not os.getenv(\"IS_TESTING\"):\n",
        "        %env GOOGLE_APPLICATION_CREDENTIALS ''"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bucket:mbsdk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "When you initialize the Vertex AI SDK for Python, you specify a Cloud Storage staging bucket. The staging bucket is where all the data associated with your dataset and model resources are retained across sessions.\n",
        "\n",
        "Set the name of your Cloud Storage bucket below. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bucket"
      },
      "outputs": [],
      "source": [
        "BUCKET_NAME = \"gs://[your-bucket-name]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "autoset_bucket"
      },
      "outputs": [],
      "source": [
        "if BUCKET_NAME == \"\" or BUCKET_NAME is None or BUCKET_NAME == \"gs://[your-bucket-name]\":\n",
        "    BUCKET_NAME = \"gs://\" + PROJECT_ID + \"aip-\" + TIMESTAMP"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_bucket"
      },
      "source": [
        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create_bucket"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l $REGION $BUCKET_NAME"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "validate_bucket"
      },
      "source": [
        "Finally, validate access to your Cloud Storage bucket by examining its contents:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "validate_bucket"
      },
      "outputs": [],
      "source": [
        "! gsutil ls -al $BUCKET_NAME"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "set_service_account"
      },
      "source": [
        "#### Service Account\n",
        "\n",
        "**If you don't know your service account**, try to get your service account using `gcloud` command by executing the second cell below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_service_account"
      },
      "outputs": [],
      "source": [
        "SERVICE_ACCOUNT = \"[your-service-account]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "autoset_service_account"
      },
      "outputs": [],
      "source": [
        "if (\n",
        "    SERVICE_ACCOUNT == \"\"\n",
        "    or SERVICE_ACCOUNT is None\n",
        "    or SERVICE_ACCOUNT == \"[your-service-account]\"\n",
        "):\n",
        "    # Get your GCP project id from gcloud\n",
        "    shell_output = !gcloud auth list 2>/dev/null\n",
        "    SERVICE_ACCOUNT = shell_output[2].replace('*', '').strip()\n",
        "    print(\"Service Account:\", SERVICE_ACCOUNT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "set_service_account:pipelines"
      },
      "source": [
        "#### Set service account access for Vertex AI Pipelines\n",
        "\n",
        "Run the following commands to grant your service account access to read and write pipeline artifacts in the bucket that you created in the previous step -- you only need to run these once per service account."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "set_service_account:pipelines"
      },
      "outputs": [],
      "source": [
        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_NAME\n",
        "\n",
        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_NAME"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "setup_vars"
      },
      "source": [
        "### Set up variables\n",
        "\n",
        "Next, set up some variables used throughout the tutorial.\n",
        "### Import libraries and define constants"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "import_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "import google.cloud.aiplatform as aip"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pipeline_constants"
      },
      "source": [
        "#### Vertex AI Pipelines constants\n",
        "\n",
        "Setup up the following constants for Vertex AI Pipelines:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pipeline_constants"
      },
      "outputs": [],
      "source": [
        "PIPELINE_ROOT = \"{}/pipeline_root/flowers\".format(BUCKET_NAME)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "additional_imports"
      },
      "source": [
        "Additional imports."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "import_pipelines:min"
      },
      "outputs": [],
      "source": [
        "import kfp"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "init_aip:mbsdk"
      },
      "source": [
        "## Initialize Vertex AI SDK for Python\n",
        "\n",
        "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "init_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "aip.init(project=PROJECT_ID, staging_bucket=BUCKET_NAME)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "define_pipeline:gcpc,automl,flowers,icn"
      },
      "source": [
        "## Define AutoML image classification model pipeline that uses components from `google_cloud_pipeline_components`\n",
        "\n",
        "Next, you define the pipeline.\n",
        "\n",
        "Create and deploy an AutoML image classification `Model` resource using a `Dataset` resource."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "define_pipeline:gcpc,automl,flowers,icn"
      },
      "outputs": [],
      "source": [
        "@kfp.dsl.pipeline(name=\"automl-image-training-v2\")\n",
        "def pipeline(project: str = PROJECT_ID, region: str = REGION):\n",
        "    from google_cloud_pipeline_components import aiplatform as gcc_aip\n",
        "    from google_cloud_pipeline_components.v1.endpoint import (EndpointCreateOp,\n",
        "                                                              ModelDeployOp)\n",
        "\n",
        "    ds_op = gcc_aip.ImageDatasetCreateOp(\n",
        "        project=project,\n",
        "        display_name=\"flowers\",\n",
        "        gcs_source=\"gs://cloud-samples-data/vision/automl_classification/flowers/all_data_v2.csv\",\n",
        "        import_schema_uri=aip.schema.dataset.ioformat.image.single_label_classification,\n",
        "    )\n",
        "\n",
        "    training_job_run_op = gcc_aip.AutoMLImageTrainingJobRunOp(\n",
        "        project=project,\n",
        "        display_name=\"train-automl-flowers\",\n",
        "        prediction_type=\"classification\",\n",
        "        model_type=\"CLOUD\",\n",
        "        dataset=ds_op.outputs[\"dataset\"],\n",
        "        model_display_name=\"train-automl-flowers\",\n",
        "        training_fraction_split=0.6,\n",
        "        validation_fraction_split=0.2,\n",
        "        test_fraction_split=0.2,\n",
        "        budget_milli_node_hours=8000,\n",
        "    )\n",
        "\n",
        "    endpoint_op = EndpointCreateOp(\n",
        "        project=project,\n",
        "        location=region,\n",
        "        display_name=\"train-automl-flowers\",\n",
        "    )\n",
        "\n",
        "    ModelDeployOp(\n",
        "        model=training_job_run_op.outputs[\"model\"],\n",
        "        endpoint=endpoint_op.outputs[\"endpoint\"],\n",
        "        automatic_resources_min_replica_count=1,\n",
        "        automatic_resources_max_replica_count=1,\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "compile_pipeline"
      },
      "source": [
        "## Compile the pipeline\n",
        "\n",
        "Next, compile the pipeline."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "compile_pipeline"
      },
      "outputs": [],
      "source": [
        "from kfp.v2 import compiler  # noqa: F811\n",
        "\n",
        "compiler.Compiler().compile(\n",
        "    pipeline_func=pipeline,\n",
        "    package_path=\"image classification_pipeline.json\".replace(\" \", \"_\"),\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "run_pipeline:automl,image"
      },
      "source": [
        "## Run the pipeline\n",
        "\n",
        "Next, run the pipeline."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "run_pipeline:automl,image"
      },
      "outputs": [],
      "source": [
        "DISPLAY_NAME = \"flowers_\" + TIMESTAMP\n",
        "\n",
        "job = aip.PipelineJob(\n",
        "    display_name=DISPLAY_NAME,\n",
        "    template_path=\"image classification_pipeline.json\".replace(\" \", \"_\"),\n",
        "    pipeline_root=PIPELINE_ROOT,\n",
        "    enable_caching=False,\n",
        ")\n",
        "\n",
        "job.run()\n",
        "\n",
        "! rm image_classification_pipeline.json"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view_pipeline_run:automl,image"
      },
      "source": [
        "Click on the generated link to see your run in the Cloud Console.\n",
        "\n",
        "<!-- It should look something like this as it is running:\n",
        "\n",
        "<a href=\"https://storage.googleapis.com/amy-jo/images/mp/automl_tabular_classif.png\" target=\"_blank\"><img src=\"https://storage.googleapis.com/amy-jo/images/mp/automl_tabular_classif.png\" width=\"40%\"/></a> -->\n",
        "\n",
        "In the UI, many of the pipeline DAG nodes will expand or collapse when you click on them. Here is a partially-expanded view of the DAG (click image to see larger version).\n",
        "\n",
        "<a href=\"https://storage.googleapis.com/amy-jo/images/mp/automl_image_classif.png\" target=\"_blank\"><img src=\"https://storage.googleapis.com/amy-jo/images/mp/automl_image_classif.png\" width=\"40%\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cleanup:pipelines"
      },
      "source": [
        "# Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial -- *Note:* this is auto-generated and not all resources may be applicable for this tutorial:\n",
        "\n",
        "- Dataset\n",
        "- Pipeline\n",
        "- Model\n",
        "- Endpoint\n",
        "- Batch Job\n",
        "- Custom Job\n",
        "- Hyperparameter Tuning Job\n",
        "- Cloud Storage Bucket"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cleanup:pipelines"
      },
      "outputs": [],
      "source": [
        "delete_dataset = True\n",
        "delete_pipeline = True\n",
        "delete_model = True\n",
        "delete_endpoint = True\n",
        "delete_batchjob = True\n",
        "delete_customjob = True\n",
        "delete_hptjob = True\n",
        "delete_bucket = True\n",
        "\n",
        "try:\n",
        "    if delete_model and \"DISPLAY_NAME\" in globals():\n",
        "        models = aip.Model.list(\n",
        "            filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
        "        )\n",
        "        model = models[0]\n",
        "        aip.Model.delete(model)\n",
        "        print(\"Deleted model:\", model)\n",
        "except Exception as e:\n",
        "    print(e)\n",
        "\n",
        "try:\n",
        "    if delete_endpoint and \"DISPLAY_NAME\" in globals():\n",
        "        endpoints = aip.Endpoint.list(\n",
        "            filter=f\"display_name={DISPLAY_NAME}_endpoint\", order_by=\"create_time\"\n",
        "        )\n",
        "        endpoint = endpoints[0]\n",
        "        endpoint.undeploy_all()\n",
        "        aip.Endpoint.delete(endpoint.resource_name)\n",
        "        print(\"Deleted endpoint:\", endpoint)\n",
        "except Exception as e:\n",
        "    print(e)\n",
        "\n",
        "if delete_dataset and \"DISPLAY_NAME\" in globals():\n",
        "    if \"image\" == \"tabular\":\n",
        "        try:\n",
        "            datasets = aip.TabularDataset.list(\n",
        "                filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
        "            )\n",
        "            dataset = datasets[0]\n",
        "            aip.TabularDataset.delete(dataset.resource_name)\n",
        "            print(\"Deleted dataset:\", dataset)\n",
        "        except Exception as e:\n",
        "            print(e)\n",
        "\n",
        "    if \"image\" == \"image\":\n",
        "        try:\n",
        "            datasets = aip.ImageDataset.list(\n",
        "                filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
        "            )\n",
        "            dataset = datasets[0]\n",
        "            aip.ImageDataset.delete(dataset.resource_name)\n",
        "            print(\"Deleted dataset:\", dataset)\n",
        "        except Exception as e:\n",
        "            print(e)\n",
        "\n",
        "    if \"image\" == \"text\":\n",
        "        try:\n",
        "            datasets = aip.TextDataset.list(\n",
        "                filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
        "            )\n",
        "            dataset = datasets[0]\n",
        "            aip.TextDataset.delete(dataset.resource_name)\n",
        "            print(\"Deleted dataset:\", dataset)\n",
        "        except Exception as e:\n",
        "            print(e)\n",
        "\n",
        "    if \"image\" == \"video\":\n",
        "        try:\n",
        "            datasets = aip.VideoDataset.list(\n",
        "                filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
        "            )\n",
        "            dataset = datasets[0]\n",
        "            aip.VideoDataset.delete(dataset.resource_name)\n",
        "            print(\"Deleted dataset:\", dataset)\n",
        "        except Exception as e:\n",
        "            print(e)\n",
        "\n",
        "try:\n",
        "    if delete_pipeline and \"DISPLAY_NAME\" in globals():\n",
        "        pipelines = aip.PipelineJob.list(\n",
        "            filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
        "        )\n",
        "        pipeline = pipelines[0]\n",
        "        aip.PipelineJob.delete(pipeline.resource_name)\n",
        "        print(\"Deleted pipeline:\", pipeline)\n",
        "except Exception as e:\n",
        "    print(e)\n",
        "\n",
        "if delete_bucket and \"BUCKET_NAME\" in globals():\n",
        "    ! gsutil rm -r $BUCKET_NAME"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "google_cloud_pipeline_components_automl_images.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}

{'fraction_non_alphanumeric': 0.1588996282527881, 'fraction_numerical': 0.002379182156133829, 'mean_word_length': 1.8899058914528812, 'pattern_counts': {'":': 588, '<': 55, '<?xml version=': 0, '>': 61, 'https://': 62, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6651011', 'n_tokens_mistral': 20732, 'n_tokens_neox': 20543, 'n_words': 4771}