<jupyter_start><jupyter_text>Notebook is a revised version of notebook from [<NAME> and <NAME>](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/matching_engine/two-tower-model-introduction.ipynb)  E2E ML on GCP: MLOps stage 6 : serving: get started with Vertex AI Matching Engine and Two Towers builtin algorithm             Run in Colab                        View on GitHub                        Run in Vertex Workbench        OverviewThis tutorial demonstrates how to use the `Vertex AI Two-Tower` built-in algorithm with `Vertex AI Matching Engine`. DatasetThis tutorial uses the `movielens_100k sample dataset` in the public bucket `gs://cloud-samples-data/vertex-ai/matching-engine/two-tower`, which was generated from the [MovieLens movie rating dataset](https://grouplens.org/datasets/movielens/100k/). For this tutorial, the data only includes the user id feature for users, and the movie id and movie title features for movies. In this example, the user is the query object and the movie is the candidate object, and each training example in the dataset contains a user and a movie they rated (we only include positive ratings in the dataset). The two-tower model will embed the user and the movie in the same embedding space, so that given a user, the model will recommend movies it thinks the user will like. ObjectiveIn this notebook, you will learn how to use the `Two-Tower` builtin algorithms for generating embeddings for a dataset, for use with generating an `Matching Engine Index`, with the `Vertex AI Matching Engine` service.This tutorial uses the following Google Cloud ML services: - `Vertex AI Two-Towers` builtin algorithm- `Vertex AI Matching Engine`- `Vertex AI Batch Prediction`The tutorial covers the following steps:1. Train the `Two-Tower` algorithm to generate embeddings (encoder) for the dataset.2. Hyperparameter tune the trained `Two-Tower` encoder.3. Make example predictions (embeddings) from then trained encoder.4. Generate embeddings using the trained `Two-Tower` builtin algorithm.5. Store embeddings to format supported by `Matching Engine`.6. Create a `Matching Engine Index` for the embeddings.7. Deploy the `Matching Engine Index` to a `Index Endpoint`.8. Make a matching engine prediction request. Costs This tutorial uses billable components of Google Cloud:* Vertex AI* Cloud StorageLearn about [Vertex AIpricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storagepricing](https://cloud.google.com/storage/pricing), and use the [PricingCalculator](https://cloud.google.com/products/calculator/)to generate a cost estimate based on your projected usage.  Set up your local development environment**If you are using Colab or Google Cloud Notebooks**, your environment already meetsall the requirements to run this notebook. You can skip this step. **Otherwise**, make sure your environment meets this notebook's requirements.You need the following:* The Google Cloud SDK* Git* Python 3* virtualenv* Jupyter notebook running in a virtual environment with Python 3The Google Cloud guide to [Setting up a Python developmentenvironment](https://cloud.google.com/python/setup) and the [Jupyterinstallation guide](https://jupyter.org/install) provide detailed instructionsfor meeting these requirements. The following steps provide a condensed set ofinstructions:1. [Install and initialize the Cloud SDK.](https://cloud.google.com/sdk/docs/)1. [Install Python 3.](https://cloud.google.com/python/setupinstalling_python)1. [Install   virtualenv](https://cloud.google.com/python/setupinstalling_and_using_virtualenv)   and create a virtual environment that uses Python 3. Activate the virtual environment.1. To install Jupyter, run `pip3 install jupyter` on thecommand-line in a terminal shell.1. To launch Jupyter, run `jupyter notebook` on the command-line in a terminal shell.1. Open this notebook in the Jupyter Notebook Dashboard.  InstallationInstall the packages required for executing this notebook.<jupyter_code>import os

# The Vertex AI Workbench Notebook product has specific requirements
IS_WORKBENCH_NOTEBOOK = os.getenv("DL_ANACONDA_HOME")
IS_USER_MANAGED_WORKBENCH_NOTEBOOK = os.path.exists(
    "/opt/deeplearning/metadata/env_version"
)

# Vertex AI Notebook requires dependencies to be installed with '--user'
USER_FLAG = ""
if IS_WORKBENCH_NOTEBOOK:
    USER_FLAG = "--user"

! pip3 install {USER_FLAG} --upgrade tensorflow -q
! pip3 install {USER_FLAG} --upgrade google-cloud-aiplatform tensorboard-plugin-profile -q
! gcloud components update --quiet<jupyter_output><empty_output><jupyter_text>Restart the kernelAfter you install the additional packages, you need to restart the notebook kernel so it can find the packages.<jupyter_code># Automatically restart kernel after installs
import os

if not os.getenv("IS_TESTING"):
    # Automatically restart kernel after installs
    import IPython

    app = IPython.Application.instance()
    app.kernel.do_shutdown(True)<jupyter_output><empty_output><jupyter_text>Before you begin  Set up your Google Cloud project**The following steps are required, regardless of your notebook environment.**1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).1. [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).1. If you are running this notebook locally, you will need to install the [Cloud SDK](https://cloud.google.com/sdk).1. Enter your project ID in the cell below. Then run the cell to make sure theCloud SDK uses the right project for all the commands in this notebook.**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands.  Set your project ID**If you do not know your project ID**, you may be able to get your project ID using `gcloud`.<jupyter_code>PROJECT_ID = "[your-project-id]"  # @param {type:"string"}
if PROJECT_ID == "" or PROJECT_ID is None or PROJECT_ID == "[your-project-id]":
    # Get your GCP project id from gcloud
    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null
    PROJECT_ID = shell_output[0]
    print("Project ID:", PROJECT_ID)
! gcloud config set project $PROJECT_ID<jupyter_output><empty_output><jupyter_text>Get your project numberNow that the project ID is set, you get your corresponding project number.<jupyter_code>shell_output = ! gcloud projects list --filter="PROJECT_ID:'{PROJECT_ID}'" --format='value(PROJECT_NUMBER)'
PROJECT_NUMBER = shell_output[0]
print("Project Number:", PROJECT_NUMBER)<jupyter_output><empty_output><jupyter_text>RegionYou can also change the `REGION` variable, which is used for operationsthroughout the rest of this notebook.  Below are regions supported for Vertex AI. We recommend that you choose the region closest to you.- Americas: `us-central1`- Europe: `europe-west4`- Asia Pacific: `asia-east1`You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations).<jupyter_code>REGION = "[your-region]"  # @param {type: "string"}

if REGION == "[your-region]":
    REGION = "us-central1"<jupyter_output><empty_output><jupyter_text>TimestampIf you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a timestamp for each instance session, and append it onto the name of resources you create in this tutorial.<jupyter_code>from datetime import datetime

TIMESTAMP = datetime.now().strftime("%Y%m%d%H%M%S")<jupyter_output><empty_output><jupyter_text>Authenticate your Google Cloud account**If you are using Vertex AI Workbench Notebooks**, your environment is alreadyauthenticated. Skip this step. **If you are using Colab**, run the cell below and follow the instructionswhen prompted to authenticate your account via oAuth.**Otherwise**, follow these steps:1. In the Cloud Console, go to the [**Create service account key**   page](https://console.cloud.google.com/apis/credentials/serviceaccountkey).2. Click **Create service account**.3. In the **Service account name** field, enter a name, and   click **Create**.4. In the **Grant this service account access to project** section, click the **Role** drop-down list. Type "Vertex AI"into the filter box, and select   **Vertex AI Administrator**. Type "Storage Object Admin" into the filter box, and select **Storage Object Admin**.5. Click *Create*. A JSON file that contains your key downloads to yourlocal environment.6. Enter the path to your service account key as the`GOOGLE_APPLICATION_CREDENTIALS` variable in the cell below and run the cell.<jupyter_code># If you are running this notebook in Colab, run this cell and follow the
# instructions to authenticate your GCP account. This provides access to your
# Cloud Storage bucket and lets you submit training jobs and prediction
# requests.

import os
import sys

# If on Vertex AI Workbench, then don't execute this code
IS_COLAB = False
if not os.path.exists("/opt/deeplearning/metadata/env_version") and not os.getenv(
    "DL_ANACONDA_HOME"
):
    if "google.colab" in sys.modules:
        IS_COLAB = True
        from google.colab import auth as google_auth

        google_auth.authenticate_user()

    # If you are running this notebook locally, replace the string below with the
    # path to your service account key and run this cell to authenticate your GCP
    # account.
    elif not os.getenv("IS_TESTING"):
        %env GOOGLE_APPLICATION_CREDENTIALS ''<jupyter_output><empty_output><jupyter_text>Create a Cloud Storage bucket**The following steps are required, regardless of your notebook environment.**Before you submit a training job for the two-tower model, you need to upload your training data and schema to Cloud Storage. Vertex AI trains the model using this input data. In this tutorial, the Two-Tower built-in algorithm also saves the trained model that results from your job in the same bucket. Using this model artifact, you can then create Vertex AI model and endpoint resources in order to serve online predictions.Set the name of your Cloud Storage bucket below. It must be unique across allCloud Storage buckets.<jupyter_code>BUCKET_NAME = "[your-bucket-name]"  # @param {type:"string"}
BUCKET_URI = f"gs://{BUCKET_NAME}"
if BUCKET_URI == "" or BUCKET_URI is None or BUCKET_URI == "gs://[your-bucket-name]":
    BUCKET_NAME = PROJECT_ID + "aip-" + TIMESTAMP
    BUCKET_URI = "gs://" + BUCKET_NAME<jupyter_output><empty_output><jupyter_text>**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket.<jupyter_code>! gsutil mb -l $REGION $BUCKET_URI<jupyter_output><empty_output><jupyter_text>Finally, validate access to your Cloud Storage bucket by examining its contents:<jupyter_code>! gsutil ls -al $BUCKET_URI<jupyter_output><empty_output><jupyter_text>Import libraries and define constants<jupyter_code>import os

from google.cloud import aiplatform

%load_ext tensorboard<jupyter_output><empty_output><jupyter_text>Initialize Vertex AI SDK for PythonInitialize the Vertex AI SDK for Python for your project and corresponding bucket.<jupyter_code>aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)<jupyter_output><empty_output><jupyter_text>Set machine typeNext, set the machine type to use for prediction.- Set the variable `DEPLOY_COMPUTE` to configure  the compute resources for the VMs you will use for for prediction. - `machine type`     - `n1-standard`: 3.75GB of memory per vCPU.     - `n1-highmem`: 6.5GB of memory per vCPU     - `n1-highcpu`: 0.9 GB of memory per vCPU - `vCPUs`: number of \[2, 4, 8, 16, 32, 64, 96 \]*Note: You may also use n2 and e2 machine types for training and deployment, but they do not support GPUs*.<jupyter_code>if os.getenv("IS_TESTING_DEPLOY_MACHINE"):
    MACHINE_TYPE = os.getenv("IS_TESTING_DEPLOY_MACHINE")
else:
    MACHINE_TYPE = "n1-standard"

VCPU = "4"
DEPLOY_COMPUTE = MACHINE_TYPE + "-" + VCPU
print("Deploy machine type", DEPLOY_COMPUTE)<jupyter_output><empty_output><jupyter_text>Introduction to Two-Tower algorithmTwo-tower models learn to represent two items of various types (such as user profiles, search queries, web documents, answer passages, or images) in the same vector space, so that similar or related items are close to each other. These two items are referred to as the query and candidate object, since when paired with a nearest neighbor search service such as Vertex Matching Engine, the two-tower model can retrieve candidate objects related to an input query object. These objects are encoded by a query and candidate encoder (the two "towers") respectively, which are trained on pairs of relevant items. This built-in algorithm exports trained query and candidate encoders as model artifacts, which can be deployed in Vertex Prediction for usage in a recommendation system.  Configure training parameters for the Two-Tower builtin algorithmThe following table shows parameters that are common to all Vertex AI Training jobs created using the `gcloud ai custom-jobs create` command. See the [official documentation](https://cloud.google.com/sdk/gcloud/reference/ai/custom-jobs/create) for all the possible arguments.| Parameter | Data type | Description | Required ||--|--|--|--|| `display-name` | string | Name of the job. | Yes || `worker-pool-spec` | string | Comma-separated list of arguments specifying a worker pool configuration (see below). | Yes || `region` | string | Region to submit the job to. | No |The `worker-pool-spec` flag can be specified multiple times, one for each worker pool. The following table shows the arguments used to specify a worker pool.| Parameter | Data type | Description | Required ||--|--|--|--|| `machine-type` | string | Machine type for the pool. See the [official documentation](https://cloud.google.com/vertex-ai/docs/training/configure-compute) for supported machines. | Yes || `replica-count` | int | The number of replicas of the machine in the pool. | No || `container-image-uri` | string | Docker image to run on each worker. | No | The following table shows the parameters for the two-tower model training job:| Parameter | Data type | Description | Required ||--|--|--|--|| `training_data_path` | string | Cloud Storage pattern where training data is stored. | Yes || `input_schema_path` | string | Cloud Storage path where the JSON input schema is stored. | Yes || `input_file_format` | string | The file format of input. Currently supports `jsonl` and `tfrecord`. | No - default is `jsonl`. || `job_dir` | string | Cloud Storage directory where the model output files will be stored. | Yes || `eval_data_path` | string | Cloud Storage pattern where eval data is stored. | No || `candidate_data_path` | string | Cloud Storage pattern where candidate data is stored. Only used for top_k_categorical_accuracy metrics. If not set, it's generated from training/eval data. | No || `train_batch_size` | int | Batch size for training. | No - Default is 100. || `eval_batch_size` | int | Batch size for evaluation. | No - Default is 100. || `eval_split` | float | Split fraction to use for the evaluation dataset, if `eval_data_path` is not provided. | No - Default is 0.2 || `optimizer` | string | Training optimizer. Lowercase string name of any TF2.3 Keras optimizer is supported ('sgd', 'nadam', 'ftrl', etc.). See [TensorFlow documentation](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers). | No - Default is 'adagrad'. || `learning_rate` | float | Learning rate for training. | No - Default is the default learning rate of the specified optimizer. || `momentum` | float | Momentum for optimizer, if specified. | No - Default is the default momentum value for the specified optimizer. || `metrics` | string | Metrics used to evaluate the model. Can be either `auc`, `top_k_categorical_accuracy` or `precision_at_1`. | No - Default is `auc`. || `num_epochs` | int | Number of epochs for training. | No - Default is 10. || `num_hidden_layers` | int | Number of hidden layers. | No || `num_nodes_hidden_layer{index}` | int | Num of nodes in hidden layer {index}. The range of index is 1 to 20. | No || `output_dim` | int | The output embedding dimension for each encoder tower of the two-tower model. | No - Default is 64. || `training_steps_per_epoch` | int | Number of steps per epoch to run the training for.  Only needed if you are using more than 1 machine or using a master machine with more than 1 gpu. | No - Default is None. || `eval_steps_per_epoch` | int | Number of steps per epoch to run the evaluation for.  Only needed if you are using more than 1 machine or using a master machine with more than 1 gpu. | No - Default is None. || `gpu_memory_alloc` | int | Amount of memory allocated per GPU (in MB). | No - Default is no limit. |<jupyter_code>DATASET_NAME = "movielens_100k"  # Change to your dataset name.

# Change to your data and schema paths. These are paths to the movielens_100k
# sample data.
TRAINING_DATA_PATH = f"gs://cloud-samples-data/vertex-ai/matching-engine/two-tower/{DATASET_NAME}/training_data/*"
INPUT_SCHEMA_PATH = f"gs://cloud-samples-data/vertex-ai/matching-engine/two-tower/{DATASET_NAME}/input_schema.json"

# URI of the two-tower training Docker image.
LEARNER_IMAGE_URI = "us-docker.pkg.dev/vertex-ai-restricted/builtin-algorithm/two-tower"

# Change to your output location.
OUTPUT_DIR = f"{BUCKET_URI}/experiment/output"

TRAIN_BATCH_SIZE = 100  # Batch size for training.
NUM_EPOCHS = 3  # Number of epochs for training.

print(f"Dataset name: {DATASET_NAME}")
print(f"Training data path: {TRAINING_DATA_PATH}")
print(f"Input schema path: {INPUT_SCHEMA_PATH}")
print(f"Output directory: {OUTPUT_DIR}")
print(f"Train batch size: {TRAIN_BATCH_SIZE}")
print(f"Number of epochs: {NUM_EPOCHS}")<jupyter_output><empty_output><jupyter_text>Train on Vertex AI Training with CPUSubmit the Two-Tower training job to `Vertex AI Training`. The following command uses a single CPU machine for training. When using single node training, `training_steps_per_epoch` and `eval_steps_per_epoch` do not need to be set.  Prepare your machine specificationNow define the machine specification for your custom hyperparameter tuning job. This tells Vertex what type of machine instance to provision for the hyperparameter tuning.  - `machine_type`: The type of GCP instance to provision -- e.g., n1-standard-8.  - `accelerator_type`: The type, if any, of hardware accelerator. In this tutorial if you previously set the variable `TRAIN_GPU != None`, you are using a GPU; otherwise you will use a CPU.  - `accelerator_count`: The number of accelerators.<jupyter_code>TRAIN_COMPUTE = "n1-standard-8"
machine_spec = {"machine_type": TRAIN_COMPUTE, "accelerator_count": 0}<jupyter_output><empty_output><jupyter_text>Prepare your disk specification(optional) Now define the disk specification for your custom hyperparameter tuning job. This tells Vertex what type and size of disk to provision in each machine instance for the hyperparameter tuning.  - `boot_disk_type`: Either SSD or Standard. SSD is faster, and Standard is less expensive. Defaults to SSD.  - `boot_disk_size_gb`: Size of disk in GB.<jupyter_code>DISK_TYPE = "pd-ssd"  # [ pd-ssd, pd-standard]
DISK_SIZE = 200  # GB

disk_spec = {"boot_disk_type": DISK_TYPE, "boot_disk_size_gb": DISK_SIZE}<jupyter_output><empty_output><jupyter_text>Define the worker pool specificationNext, you define the worker pool specification for your custom hyperparameter tuning job. The worker pool specification will consist of the following:- `replica_count`: The number of instances to provision of this machine type.- `machine_spec`: The hardware specification.- `disk_spec` : (optional) The disk storage specification.- `container_spec`: The training container containing the training package.Let's dive deeper now into the container specification:- `image_uri`: The training image.- `command`: The command to invoke in the training image. Defaults to the command entry point specified for the training image.- `args`: The command line arguments to pass to the corresponding command entry point in training image.<jupyter_code>JOB_NAME = "twotowers_cpu_" + TIMESTAMP
MODEL_DIR = "{}/{}".format(BUCKET_URI, JOB_NAME)

CMDARGS = [
    f"--training_data_path={TRAINING_DATA_PATH}",
    f"--input_schema_path={INPUT_SCHEMA_PATH}",
    f"--job-dir={OUTPUT_DIR}",
    f"--train_batch_size={TRAIN_BATCH_SIZE}",
    f"--num_epochs={NUM_EPOCHS}",
]

worker_pool_spec = [
    {
        "replica_count": 1,
        "machine_spec": machine_spec,
        "disk_spec": disk_spec,
        "container_spec": {
            "image_uri": LEARNER_IMAGE_URI,
            "command": [],
            "args": CMDARGS,
        },
    }
]<jupyter_output><empty_output><jupyter_text>Create a custom jobUse the class `CustomJob` to create a custom job, such as for hyperparameter tuning, with the following parameters:- `display_name`: A human readable name for the custom job.- `worker_pool_specs`: The specification for the corresponding VM instances.<jupyter_code>job = aiplatform.CustomJob(
    display_name="twotower_cpu_" + TIMESTAMP, worker_pool_specs=worker_pool_spec
)<jupyter_output><empty_output><jupyter_text>Execute the custom jobNext, execute your custom job using the method `run()`.<jupyter_code>job.run()<jupyter_output><empty_output><jupyter_text>View outputAfter the job finishes successfully, you can view the output directory.<jupyter_code>! gsutil ls {OUTPUT_DIR}

! gsutil rm -rf {OUTPUT_DIR}/*<jupyter_output><empty_output><jupyter_text>Train on Vertex AI Training with GPUNext, train the Two Tower model using a GPU.<jupyter_code>JOB_NAME = "twotowers_gpu_" + TIMESTAMP
MODEL_DIR = "{}/{}".format(BUCKET_URI, JOB_NAME)

TRAIN_COMPUTE = "n1-highmem-4"
TRAIN_GPU = "NVIDIA_TESLA_K80"
machine_spec = {
    "machine_type": TRAIN_COMPUTE,
    "accelerator_type": TRAIN_GPU,
    "accelerator_count": 1,
}

CMDARGS = [
    f"--training_data_path={TRAINING_DATA_PATH}",
    f"--input_schema_path={INPUT_SCHEMA_PATH}",
    f"--job-dir={OUTPUT_DIR}",
    "--training_steps_per_epoch=1500",
    "--eval_steps_per_epoch=1500",
]

worker_pool_spec = [
    {
        "replica_count": 1,
        "machine_spec": machine_spec,
        "disk_spec": disk_spec,
        "container_spec": {
            "image_uri": LEARNER_IMAGE_URI,
            "command": [],
            "args": CMDARGS,
        },
    }
]<jupyter_output><empty_output><jupyter_text>Create and execute the custom jobNext, create and execute the custom job.<jupyter_code>job = aiplatform.CustomJob(
    display_name="twotower_cpu_" + TIMESTAMP, worker_pool_specs=worker_pool_spec
)

job.run()<jupyter_output><empty_output><jupyter_text>View outputAfter the job finishes successfully, you can view the output directory.<jupyter_code>! gsutil ls {OUTPUT_DIR}

! gsutil rm -rf {OUTPUT_DIR}/*<jupyter_output><empty_output><jupyter_text>Train on Vertex AI Training with TFRecordsNext, train the Two Tower model using TFRecords<jupyter_code>TRAINING_DATA_PATH = f"gs://cloud-samples-data/vertex-ai/matching-engine/two-tower/{DATASET_NAME}/tfrecord/*"

JOB_NAME = "twotowers_tfrec_" + TIMESTAMP
MODEL_DIR = "{}/{}".format(BUCKET_URI, JOB_NAME)

TRAIN_COMPUTE = "n1-standard-8"
machine_spec = {"machine_type": TRAIN_COMPUTE, "accelerator_count": 0}

CMDARGS = [
    f"--training_data_path={TRAINING_DATA_PATH}",
    f"--input_schema_path={INPUT_SCHEMA_PATH}",
    f"--job-dir={OUTPUT_DIR}",
    f"--train_batch_size={TRAIN_BATCH_SIZE}",
    f"--num_epochs={NUM_EPOCHS}",
    "--input_file_format=tfrecord",
]

worker_pool_spec = [
    {
        "replica_count": 1,
        "machine_spec": machine_spec,
        "disk_spec": disk_spec,
        "container_spec": {
            "image_uri": LEARNER_IMAGE_URI,
            "command": [],
            "args": CMDARGS,
        },
    }
]<jupyter_output><empty_output><jupyter_text>Create and execute the custom jobNext, create and execute the custom job.<jupyter_code>job = aiplatform.CustomJob(
    display_name="twotower_cpu_" + TIMESTAMP, worker_pool_specs=worker_pool_spec
)

job.run()<jupyter_output><empty_output><jupyter_text>View outputAfter the job finishes successfully, you can view the output directory.<jupyter_code>! gsutil ls {OUTPUT_DIR}

! gsutil rm -rf {OUTPUT_DIR}<jupyter_output><empty_output><jupyter_text>TensorboardWhen the training starts, you can view the logs in TensorBoard. Colab users can use the TensorBoard widget below: For Workbench AI Notebooks users, the TensorBoard widget above won't work. We recommend you to launch TensorBoard through the Cloud Shell.1. In your Cloud Shell, launch Tensorboard on port 8080:    ```    export TENSORBOARD_DIR=gs://xxxxx/tensorboard    tensorboard --logdir=${TENSORBOARD_DIR} --port=8080    ```2. Click the "Web Preview" button at the top-right of the Cloud Shell window (looks like an eye in a rectangle). 3. Select "Preview on port 8080". This should launch the TensorBoard webpage in a new tab in your browser.After the job finishes successfully, you can view the output directory:<jupyter_code>try:
    TENSORBOARD_DIR = os.path.join(OUTPUT_DIR, "tensorboard")
    %tensorboard --logdir {TENSORBOARD_DIR}
except Exception as e:
    print(e)<jupyter_output><empty_output><jupyter_text>Hyperparameter tuningYou may want to optimize the hyperparameters used during training to improve your model's accuracy and performance. For this example, the following command runs a Vertex AI hyperparameter tuning job with 8 trials that attempts to maximize the validation AUC metric. The hyperparameters it optimizes are the number of hidden layers, the size of the hidden layers, and the learning rate.Learn more about [Hyperparameter tuning overview](https://cloud.google.com/vertex-ai/docs/training/hyperparameter-tuning-overview).<jupyter_code>from google.cloud.aiplatform import hyperparameter_tuning as hpt

hpt_job = aiplatform.HyperparameterTuningJob(
    display_name="twotowers_" + TIMESTAMP,
    custom_job=job,
    metric_spec={
        "val_auc": "maximize",
    },
    parameter_spec={
        "learning_rate": hpt.DoubleParameterSpec(min=0.0001, max=0.1, scale="log"),
        "num_hidden_layers": hpt.IntegerParameterSpec(min=0, max=2, scale="linear"),
        "num_nodes_hidden_layer1": hpt.IntegerParameterSpec(
            min=1, max=128, scale="log"
        ),
        "num_nodes_hidden_layer2": hpt.IntegerParameterSpec(
            min=1, max=128, scale="log"
        ),
    },
    search_algorithm=None,
    max_trial_count=8,
    parallel_trial_count=1,
)<jupyter_output><empty_output><jupyter_text>Run the hyperparameter tuning jobUse the `run()` method to execute the hyperparameter tuning job.<jupyter_code>hpt_job.run()<jupyter_output><empty_output><jupyter_text>Display the hyperparameter tuning job trial resultsAfter the hyperparameter tuning job has completed, the property `trials` will return the results for each trial.<jupyter_code>print(hpt_job.trials)<jupyter_output><empty_output><jupyter_text>Best trialNow look at which trial was the best:<jupyter_code>best = (None, None, None, 0.0)
for trial in hpt_job.trials:
    # Keep track of the best outcome
    if float(trial.final_measurement.metrics[0].value) > best[3]:
        try:
            best = (
                trial.id,
                float(trial.parameters[0].value),
                float(trial.parameters[1].value),
                float(trial.final_measurement.metrics[0].value),
            )
        except:
            best = (
                trial.id,
                float(trial.parameters[0].value),
                None,
                float(trial.final_measurement.metrics[0].value),
            )

print(best)<jupyter_output><empty_output><jupyter_text>Delete the hyperparameter tuning jobThe method 'delete()' will delete the hyperparameter tuning job.<jupyter_code>hpt_job.delete()<jupyter_output><empty_output><jupyter_text>View outputAfter the job finishes successfully, you can view the output directory.<jupyter_code>BEST_MODEL = OUTPUT_DIR + "/trial_" + best[0]
! gsutil ls {BEST_MODEL}<jupyter_output><empty_output><jupyter_text>Upload the model to `Vertex AI Model` resourceYour training job will export two TF SavedModels under `gs:///query_model` and `gs:///candidate_model`. These exported models can be used for online or batch prediction in Vertex Prediction. First, import the query (or candidate) model using the `upload()` method, with the following parameters:- `display_name`: A human readable name for the model resource.- `artifact_uri`: The Cloud Storage location of the model artifacts.- `serving_container_image_uri`: The deployment container. In this tutorial, you use the prebuilt Two-Tower deployment container.- `serving_container_health_route`: The URL for the service to periodically ping for a response to verify that the serving binary is running. For Two-Towers, this will be /v1/models/\[model_name\].- `serving_container_predict_route`: The URL for the service to periodically ping for a response to verify that the serving binary is running. For Two-Towers, this will be /v1/models/\[model_name\]:predict.- `serving_container_environment_variables`: Preset environment variables to pass into the deployment container.*Note:* The underlying deployment container is built on TensorFlow Serving.<jupyter_code># The following imports the query (user) encoder model.
MODEL_TYPE = "query"
# Use the following instead to import the candidate (movie) encoder model.
# MODEL_TYPE = 'candidate'

DISPLAY_NAME = f"{DATASET_NAME}_{MODEL_TYPE}"  # The display name of the model.
MODEL_NAME = f"{MODEL_TYPE}_model"  # Used by the deployment container.

model = aiplatform.Model.upload(
    display_name=DISPLAY_NAME,
    artifact_uri=BEST_MODEL,
    serving_container_image_uri="us-central1-docker.pkg.dev/cloud-ml-algos/two-tower/deploy",
    serving_container_health_route=f"/v1/models/{MODEL_NAME}",
    serving_container_predict_route=f"/v1/models/{MODEL_NAME}:predict",
    serving_container_environment_variables={
        "MODEL_BASE_PATH": "$(AIP_STORAGE_URI)",
        "MODEL_NAME": MODEL_NAME,
    },
)<jupyter_output><empty_output><jupyter_text>Deploy the model to `Vertex AI Endpoint`Deploying the `Vertex AI Model` resoure to a `Vertex AI Endpoint` for online predictions:1. Create an `Endpoint` resource exposing an external interface to users consuming the model. 2. After the `Endpoint` is ready, deploy one or more instances of a model to the `Endpoint`. The deployed model runs the custom container image running Two-Tower encoder to serve embeddings.Refer to Vertex AI Predictions guide to [Deploy a model using the Vertex AI API](https://cloud.google.com/vertex-ai/docs/predictions/deploy-model-api) for more information about the APIs used in the following cells.  Create a `Vertex AI Endpoint`Next, you create the `Vertex AI Endpoint`, from which you subsequently deploy your `Vertex AI Model` resource to.<jupyter_code>endpoint = aiplatform.Endpoint.create(display_name=DATASET_NAME)<jupyter_output><empty_output><jupyter_text>Deploying `Model` resources to an `Endpoint` resource.You can deploy one of more `Vertex AI Model` resource instances to the same endpoint. Each `Vertex AI Model` resource that is deployed will have its own deployment container for the serving binary. In the next example, you deploy the `Vertex AI Model` resource to a `Vertex AI Endpoint` resource. The `Vertex AI Model` resource already has defined for it the deployment container image. To deploy, you specify the following additional configuration settings:- The machine type.- The (if any) type and number of GPUs.- Static, manual or auto-scaling of VM instances.In this example, you deploy the model with the minimal amount of specified parameters, as follows:- `model`: The `Model` resource.- `deployed_model_displayed_name`: The human readable name for the deployed model instance.- `machine_type`: The machine type for each VM instance.Do to the requirements to provision the resource, this may take upto a few minutes.<jupyter_code>response = endpoint.deploy(
    model=model,
    deployed_model_display_name=DISPLAY_NAME,
    machine_type=DEPLOY_COMPUTE,
    traffic_split={"0": 100},
)

print(endpoint)<jupyter_output><empty_output><jupyter_text>Creating embeddingsNow that you have deployed the query/candidate encoder model on `Vertex AI Prediction`, you can call the model to generate embeddings for new data.  Make an online prediction with SDK[Online prediction](https://cloud.google.com/vertex-ai/docs/predictions/online-predictions-custom-models) is used to synchronously query a model on a small batch of instances with minimal latency. The following function calls the deployed model using Vertex AI SDK for Python.The input data you want predicted embeddings on should be provided as a stringified JSON in the `data` field. Note that you should also provide a unique `key` field (of type str) for each input instance so that you can associate each output embedding with its corresponding input.<jupyter_code># Input items for the query model:
input_items = [
    {"data": '{"user_id": ["1"]}', "key": "key1"},
    {"data": '{"user_id": ["2"]}', "key": "key2"},
]

# Input items for the candidate model:
# input_items = [{
#     'data' : '{"movie_id": ["1"], "movie_title": ["fake title"]}',
#     'key': 'key1'
# }]

encodings = endpoint.predict(input_items)
print(f"Number of encodings: {len(encodings.predictions)}")
print(encodings.predictions[0]["encoding"])<jupyter_output><empty_output><jupyter_text>Make an online prediction with `gcloud`You can also do online prediction using the gcloud CLI.<jupyter_code>import json

request = json.dumps({"instances": input_items})
with open("request.json", "w") as writer:
    writer.write(f"{request}\n")

ENDPOINT_ID = endpoint.resource_name

! gcloud ai endpoints predict {ENDPOINT_ID} \
  --region={REGION} \
  --json-request=request.json<jupyter_output><empty_output><jupyter_text>Make a batch prediction[Batch prediction](https://cloud.google.com/vertex-ai/docs/predictions/batch-predictions) is used to asynchronously make predictions on a batch of input data.  This is recommended if you have a large input size and do not need an immediate response, such as getting embeddings for candidate objects in order to create an index for a nearest neighbor search service such as [Vertex Matching Engine](https://cloud.google.com/vertex-ai/docs/matching-engine/overview). Create the batch input fileNext, you generate the batch input file to generate embeddings for the dataset, which you subsequently use to create an index with `Vertex AI Matching Engine`. In this example, the dataset contains a 1000 unique identifiers (0...999). You will use the trained encoder to generate a predicted embedding for each unique identifier.The input data needs to be on Cloud Storage and in JSONL format. You can use the sample query object file provided below. Like with online prediction, it's recommended to have the `key` field so that you can associate each output embedding with its corresponding input.<jupyter_code>QUERY_EMBEDDING_PATH = f"{BUCKET_URI}/embeddings/train.jsonl"

import tensorflow as tf

with tf.io.gfile.GFile(QUERY_EMBEDDING_PATH, "w") as f:
    for i in range(0, 1000):
        query = {"data": '{"user_id": ["' + str(i) + '"]}', "key": f"key{i}"}
        f.write(json.dumps(query) + "\n")

print("\nNumber of embeddings: ")
! gsutil cat {QUERY_EMBEDDING_PATH} | wc -l<jupyter_output><empty_output><jupyter_text>Send the prediction requestTo make a batch prediction request, call the model object's `batch_predict` method with the following parameters: - `instances_format`: The format of the batch prediction request file: "jsonl", "csv", "bigquery", "tf-record", "tf-record-gzip" or "file-list"- `prediction_format`: The format of the batch prediction response file: "jsonl", "csv", "bigquery", "tf-record", "tf-record-gzip" or "file-list"- `job_display_name`: The human readable name for the prediction job.- `gcs_source`: A list of one or more Cloud Storage paths to your batch prediction requests.- `gcs_destination_prefix`: The Cloud Storage path that the service will write the predictions to.- `model_parameters`: Additional filtering parameters for serving prediction results.- `machine_type`: The type of machine to use for training.- `accelerator_type`: The hardware accelerator type.- `accelerator_count`: The number of accelerators to attach to a worker replica.- `starting_replica_count`: The number of compute instances to initially provision.- `max_replica_count`: The maximum number of compute instances to scale to. In this tutorial, only one instance is provisioned. Compute instance scalingYou can specify a single instance (or node) to process your batch prediction request. This tutorial uses a single node, so the variables `MIN_NODES` and `MAX_NODES` are both set to `1`.If you want to use multiple nodes to process your batch prediction request, set `MAX_NODES` to the maximum number of nodes you want to use. Vertex AI autoscales the number of nodes used to serve your predictions, up to the maximum number you set. Refer to the [pricing page](https://cloud.google.com/vertex-ai/pricingprediction-prices) to understand the costs of autoscaling with multiple nodes.<jupyter_code>MIN_NODES = 1
MAX_NODES = 4

batch_predict_job = model.batch_predict(
    job_display_name=f"batch_predict_{DISPLAY_NAME}",
    gcs_source=[QUERY_EMBEDDING_PATH],
    gcs_destination_prefix=f"{BUCKET_URI}/embeddings/output",
    machine_type=DEPLOY_COMPUTE,
    starting_replica_count=MIN_NODES,
    max_replica_count=MAX_NODES,
)<jupyter_output><empty_output><jupyter_text>Get the predicted embeddingsNext, get the results from the completed batch prediction job.The results are written to the Cloud Storage output bucket you specified in the batch prediction request. You call the method `iter_outputs()` to get a list of each Cloud Storage file generated with the results. Each file contains one or more prediction requests in a JSON format:- `instance`: The prediction request.- `prediction`: The prediction response.<jupyter_code>bp_iter_outputs = batch_predict_job.iter_outputs()

prediction_results = list()
for blob in bp_iter_outputs:
    if blob.name.split("/")[-1].startswith("prediction"):
        prediction_results.append(blob.name)

result_files = []
for prediction_result in prediction_results:
    result_file = f"gs://{bp_iter_outputs.bucket.name}/{prediction_result}"
    result_files.append(result_file)

print(result_files)<jupyter_output><empty_output><jupyter_text>Save the embeddings in JSONL formatNext, you store the predicted embeddings as a JSONL formatted file. Each embedding is stored as:    { 'id': .., 'embedding': [ ... ] }    The format of the embeddings for the index can be in either CSV, JSON, or Avro format.Learn more about [Embedding Formats for Indexing](https://cloud.google.com/vertex-ai/docs/matching-engine/using-matching-enginejson)<jupyter_code>embeddings = []
for result_file in result_files:

    with tf.io.gfile.GFile(result_file, "r") as f:
        instances = list(f)

    for instance in instances:
        instance = instance.replace('\\"', "'")
        result = json.loads(instance)
        prediction = result["prediction"]
        key = prediction["key"][3:]
        encoding = prediction["encoding"]

        embedding = {"id": key, "embedding": encoding}
        embeddings.append(embedding)

print("Number of embeddings", len(embeddings))
print("Encoding Dimensions", len(embeddings[0]["embedding"]))
print("Example embedding", embeddings[0])

with open("embeddings.json", "w") as f:
    for i in range(len(embeddings)):
        f.write(json.dumps(embeddings[i]).replace('"', "'"))
        f.write("\n")

! head -n 2 embeddings.json<jupyter_output><empty_output><jupyter_text>Store the JSONL formatted embeddings in Cloud StorageNext, you upload the training data to your Cloud Storage bucket.<jupyter_code>EMBEDDINGS_URI = f"{BUCKET_URI}/embeddings/twotower/"
! gsutil cp embeddings.json {EMBEDDINGS_URI}<jupyter_output><empty_output><jupyter_text>Create Matching Engine IndexNext, you create the index for your embeddings. Currently, two indexing algorithms are supported:- `create_tree_ah_index()`:  Shallow tree + Asymmetric hashing.- `create_brute_force_index()`: Linear search.In this tutorial, you use the `create_tree_ah_index()`for production scale. The method is called with the following parameters:- `display_name`: A human readable name for the index.- `contents_delta_uri`: A Cloud Storage location for the embeddings, which are either to be inserted, updated or deleted.- `dimensions`: The number of dimensions of the input vector- `approximate_neighbors_count`: (for Tree AH) The default number of neighbors to find via approximate search before exact reordering is performed. Exact reordering is a procedure where results returned by an approximate search algorithm are reordered via a more expensive distance computation.- `distance_measure_type`: The distance measure used in nearest neighbor search.    - `SQUARED_L2_DISTANCE`: Euclidean (L2) Distance    - `L1_DISTANCE`: Manhattan (L1) Distance    - `COSINE_DISTANCE`: Cosine Distance. Defined as 1 - cosine similarity.    - `DOT_PRODUCT_DISTANCE`: Default value. Defined as a negative of the dot product.- `description`: A human readble description of the index.- `labels`: User metadata in the form of a dictionary.- `leaf_node_embedding_count`: Number of embeddings on each leaf node. The default value is 1000 if not set.- `leaf_nodes_to_search_percent`: The default percentage of leaf nodes that any query may be searched. Must be in range 1-100, inclusive. The default value is 10 (means 10%) if not set.This may take upto 30 minutes.Learn more about [Configuring Matching Engine Indexes](https://cloud.google.com/vertex-ai/docs/matching-engine/configuring-indexes).<jupyter_code>DIMENSIONS = len(embeddings[0]["embedding"])
DISPLAY_NAME = "movies"

tree_ah_index = aiplatform.MatchingEngineIndex.create_tree_ah_index(
    display_name=DISPLAY_NAME,
    contents_delta_uri=EMBEDDINGS_URI,
    dimensions=DIMENSIONS,
    approximate_neighbors_count=50,
    distance_measure_type="DOT_PRODUCT_DISTANCE",
    description="Two tower generated embeddings",
    labels={"label_name": "label_value"},
    # TreeAH specific parameters
    leaf_node_embedding_count=100,
    leaf_nodes_to_search_percent=7,
)

INDEX_RESOURCE_NAME = tree_ah_index.resource_name
print(INDEX_RESOURCE_NAME)<jupyter_output><empty_output><jupyter_text>Setup VPC peering networkTo use a `Matching Engine Index`, you setup a VPC peering network between your project and the `Vertex AI Matching Engine` service project. This eliminates additional hops in network traffic and allows using efficient gRPC protocol.Learn more about [VPC peering](https://cloud.google.com/vertex-ai/docs/general/vpc-peering).**IMPORTANT: you can only setup one VPC peering to servicenetworking.googleapis.com per project.**  Create VPC peering for default networkFor simplicity, we setup VPC peering to the default network. You can create a different network for your project.If you setup VPC peering with any other network, make sure that the network already exists and that your VM is running on that network.<jupyter_code># This is for display only; you can name the range anything.
PEERING_RANGE_NAME = "vertex-ai-prediction-peering-range"
NETWORK = "default"

# NOTE: `prefix-length=16` means a CIDR block with mask /16 will be
# reserved for use by Google services, such as Vertex AI.
! gcloud compute addresses create $PEERING_RANGE_NAME \
  --global \
  --prefix-length=16 \
  --description="peering range for Google service" \
  --network=$NETWORK \
  --purpose=VPC_PEERING<jupyter_output><empty_output><jupyter_text>Create the VPC connectionNext, create the connection for VPC peering.*Note:* If you get a PERMISSION DENIED, you may not have the neccessary role 'Compute Network Admin' set for your default service account. In the Cloud Console, do the following steps.1. Goto `IAM & Admin`2. Find your service account.3. Click edit icon.4. Select `Add Another Role`.5. Enter 'Compute Network Admin'.6. Select `Save`<jupyter_code>! gcloud services vpc-peerings connect \
  --service=servicenetworking.googleapis.com \
  --network=$NETWORK \
  --ranges=$PEERING_RANGE_NAME \
  --project=$PROJECT_ID<jupyter_output><empty_output><jupyter_text>Check the status of your peering connections.<jupyter_code>! gcloud compute networks peerings list --network $NETWORK<jupyter_output><empty_output><jupyter_text>Construct the full network nameYou need to have the full network resource name when you subsequently create an `Matching Engine Index Endpoint` resource for VPC peering.<jupyter_code>full_network_name = f"projects/{PROJECT_NUMBER}/global/networks/{NETWORK}"<jupyter_output><empty_output><jupyter_text>Create an IndexEndpoint with VPC NetworkNext, you create a `Matching Engine Index Endpoint`, similar to the concept of creating a `Private Endpoint` for prediction with a peer-to-peer network.To create the `Index Endpoint` resource, you call the method `create()` with the following parameters:- `display_name`: A human readable name for the `Index Endpoint`.- `description`: A description for the `Index Endpoint`.- `network`: The VPC network resource name.<jupyter_code>index_endpoint = aiplatform.MatchingEngineIndexEndpoint.create(
    display_name="index_endpoint_for_demo",
    description="index endpoint description",
    network=full_network_name,
)

INDEX_ENDPOINT_NAME = index_endpoint.resource_name
print(INDEX_ENDPOINT_NAME)<jupyter_output><empty_output><jupyter_text>Deploy the `Matching Engine Index` to the `Index Endpoint` resourceNext, deploy your index to the `Index Endpoint` using the method `deploy_index()` with the following parameters:- `display_name`: A human readable name for the deployed index.- `index`: Your index.- `deployed_index_id`: A user assigned identifier for the deployed index.- `machine_type`: (optional) The VM instance type.- `min_replica_count`: (optional) Minimum number of VM instances for auto-scaling.- `max_replica_count`: (optional) Maximum number of VM instances for auto-scaling.Learn more about [Machine resources for Index Endpoint](https://cloud.google.com/vertex-ai/docs/reference/rest/v1/projects.locations.indexEndpointsDeployedIndex)<jupyter_code>DEPLOYED_INDEX_ID = "tree_ah_twotower_deployed_" + TIMESTAMP

MIN_NODES = 1
MAX_NODES = 2
DEPLOY_COMPUTE = "n1-standard-16"

index_endpoint.deploy_index(
    display_name="deployed_index_for_demo",
    index=tree_ah_index,
    deployed_index_id=DEPLOYED_INDEX_ID,
    machine_type=DEPLOY_COMPUTE,
    min_replica_count=MIN_NODES,
    max_replica_count=MAX_NODES,
)

print(index_endpoint.deployed_indexes)<jupyter_output><empty_output><jupyter_text>Create and execute an online queryNow that your index is deployed, you can make queries.First, you construct a vector `query` using synthetic data, to use as the example to return matches for.Next, you make the matching request using the method `match()`, with the following parameters:- `deployed_index_id`:  The identifier of the deployed index.- `queries`: A list of queries (instances).- `num_neighbors`: The number of closest matches to return.<jupyter_code># The number of nearest neighbors to be retrieved from database for each query.
NUM_NEIGHBOURS = 10

# Test query
queries = [embeddings[0]["embedding"], embeddings[1]["embedding"]]

matches = index_endpoint.match(
    deployed_index_id=DEPLOYED_INDEX_ID, queries=queries, num_neighbors=NUM_NEIGHBOURS
)

for instance in matches:
    print("INSTANCE")
    for match in instance:
        print(match)<jupyter_output><empty_output><jupyter_text>Cleaning upTo clean up all Google Cloud resources used in this project, you can [delete the Google Cloudproject](https://cloud.google.com/resource-manager/docs/creating-managing-projectsshutting_down_projects) you used for the tutorial.Otherwise, you can delete the individual resources you created in this tutorial:<jupyter_code># Delete endpoint resource
endpoint.delete(force=True)

# Delete model resource
model.delete()

# Force undeployment of indexes and delete endpoint
try:
    index_endpoint.delete(force=True)
except Exception as e:
    print(e)

# Delete indexes
try:
    tree_ah_index.delete()
    brute_force_index.delete()
except Exception as e:
    print(e)

# Delete Cloud Storage objects that were created
delete_bucket = False
if delete_bucket or os.getenv("IS_TESTING"):
    ! gsutil -m rm -r $OUTPUT_DIR<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09024151436031332, 'fraction_numerical': 0.0054667101827676244, 'mean_word_length': 5.0042865890998165, 'pattern_counts': {'":': 56, '<': 211, '<?xml version=': 0, '>': 213, 'https://': 30, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '52', 'dir': 'jupyter-structured-clean-dedup', 'id': '480833', 'license': 'Apache-2.0', 'n_tokens_mistral': 14410, 'n_tokens_neox': 13910, 'n_words': 5135, 'path': 'notebooks/community/ml_ops/stage6/get_started_with_matching_engine_twotowers.ipynb', 'repo_name': 'nayaknishant/vertex-ai-samples', 'size': '49028'}