<jupyter_start><jupyter_text>AI Platform Prediction Load Testing using LocustThis Notebook demonstrates how to perform load testing of AI Platform Prediction using [Locust](https://locust.io). The Notebook covers the following steps:1. Creating custom log-based metrics in Cloud Monitoring for the Locust logs2. Creating a Cloud Monitoring Dashboard to display the AI Platform and custom metrics3. Deploying Locust to GKE clustor4. Configuring Lucost test5. Runing the Lucost load test  SetupThis notebook was tested on **AI Platform Notebooks** using the standard TF 2.2 image.  Install required packages<jupyter_code>%pip install -q -U locust google-cloud-monitoring google-cloud-logging google-cloud-monitoring-dashboards
# Automatically restart kernel after installs
import IPython
app = IPython.Application.instance()
app.kernel.do_shutdown(True)<jupyter_output><empty_output><jupyter_text>Import libraries<jupyter_code>import base64
import os
import time
import datetime
import json
import requests

import numpy as np
import pandas as pd

import google.auth

from typing import List

from google.api_core.exceptions import GoogleAPICallError 

from google.cloud import logging_v2
from google.cloud.logging_v2 import MetricsServiceV2Client
from google.cloud.logging_v2 import LoggingServiceV2Client

from google.cloud.monitoring_dashboard.v1.types import Dashboard
from google.cloud.monitoring_dashboard.v1 import DashboardsServiceClient
from google.cloud.monitoring_v3 import MetricServiceClient
from google.cloud.monitoring_v3.query import Query
from google.cloud.monitoring_v3.types import TimeInterval

from google.protobuf.json_format import ParseDict<jupyter_output><empty_output><jupyter_text>Configure GCP environment settings<jupyter_code>PROJECT_ID = '[your-google-project-id]' # Set your project Id
BUCKET = '[your-bucket-name]' # Set your bucket name Id
REGION = '[your-region]'  # Set your region for deploying the model
GKE_CLUSTER_NAME = '[your-gke-cluster-name]' # Set the GKE cluster name
GKE_CLUSTER_ZONE = '[your-gke-cluster-zone]' # Set the GKE cluster zone
MODEL_NAME = 'resnet_classifier'
MODEL_VERSION = 'v1'
IMAGES_FOLDER = 'test_images'
GCS_LOCUST_TEST_CONFIG_DIR = 'gs://{}/locust-test'.format(BUCKET)<jupyter_output><empty_output><jupyter_text>1. Creating Custom Log-based Metrics in Cloud MonitoringIn this section of the notebook you will use the [Python Cloud Logging client library](https://googleapis.dev/python/logging/latest/v2.html) to create a set of custom log-based metrics. The metrics are based on the log entries generated by the example locustfile script. The script writes the log entries into the *Cloud Logging* log named `locust`.Each log entry includes a set of key value pairs encoded as the JSON payload type. The metrics are based on the subset of keys from the log entry.Key | Value----|------test_id | An ID of a testmodel | An AI Platform Prediction Model namemodel_version | An AI Platform Prediction Model versionlatency | A 95 percentile response time, which is calculated over a 10 sliding second windownum_requests | A total number of requests since the test startednum_failures | A total number of requests since the test starteduser_count | A number of simulated users rps | A current requests per secondRefer to the [Cloud Logging API reference](https://googleapis.dev/python/logging/latest/v2.html) for more information about the API.  1.1. Define a helper function that creates a custom log metric<jupyter_code>def create_locust_metric(
    metric_name:str,
    log_path:str, 
    value_field:str,  
    bucket_bounds:List[int]):
    
    metric_path = logging_client.metric_path(PROJECT_ID, metric_name)
    log_entry_filter = 'resource.type=global AND logName={}'.format(log_path)
    
    metric_descriptor = {
        'metric_kind': 'DELTA',
        'value_type': 'DISTRIBUTION',
        'labels': [
            {
                'key': 'test_id',
                'value_type': 'STRING'
            },
            {
                'key': 'signature',
                'value_type': 'STRING'
            }
        ]
    }
    
    bucket_options = {
        'explicit_buckets': {
            'bounds': bucket_bounds
        }
    }
    
    value_extractor = 'EXTRACT(jsonPayload.{})'.format(value_field)
    label_extractors = {
        'test_id': 'EXTRACT(jsonPayload.test_id)',
        'signature': 'EXTRACT(jsonPayload.signature)'
    }
    
    metric = logging_v2.types.LogMetric(
        name=metric_name,
        filter=log_entry_filter,
        value_extractor=value_extractor,
        bucket_options=bucket_options,
        label_extractors=label_extractors,
        metric_descriptor=metric_descriptor,
    )
    
    try:
        logging_client.get_log_metric(metric_path)
        print('Metric: {} already exists'.format(metric_path))
    except:
        logging_client.create_log_metric(parent, metric)
        print('Created metric {}'.format(metric_path))<jupyter_output><empty_output><jupyter_text>1.2. Create a logging client.<jupyter_code>log_name = 'locust'

creds , _ = google.auth.default()
logging_client = MetricsServiceV2Client(credentials=creds)

parent = logging_client.project_path(PROJECT_ID)
log_path = LoggingServiceV2Client.log_path(PROJECT_ID, log_name)<jupyter_output><empty_output><jupyter_text>1.3. Create metrics to track Locust logs<jupyter_code># user count metric
metric_name = 'locust_users'
value_field = 'user_count'
bucket_bounds = [1, 16, 32, 64, 128]

create_locust_metric(metric_name, log_path, value_field, bucket_bounds)

# latency metric
metric_name = 'locust_latency'
value_field = 'latency'
bucket_bounds = [1, 50, 100, 200, 500]

create_locust_metric(metric_name, log_path, value_field, bucket_bounds)

# failure count metric
metric_name = 'num_failures'
value_field = 'num_failures'
bucket_bounds = [1, 1000]

create_locust_metric(metric_name, log_path, value_field, bucket_bounds)

# request count metric
metric_name = 'num_requests'
value_field = 'num_requests'
bucket_bounds = [1, 1000]

create_locust_metric(metric_name, log_path, value_field, bucket_bounds)<jupyter_output><empty_output><jupyter_text>1.4. List metrics<jupyter_code>metrics = logging_client.list_log_metrics(parent)

if not list(metrics):
    print("There are not any log based metrics defined in the the project")
else:
    for element in logging_client.list_log_metrics(parent):
        print(element.metric_descriptor.name)<jupyter_output><empty_output><jupyter_text>2. Creating the Cloud Monitoring dashboardThe`dashboard_template` folder contains an example monitoring dashboard template that combines standard AI Platform Prediction metrics with log-based metrics defined in the previous steps. You can use [Python Client for Cloud Monitoring Dashboards API](https://googleapis.dev/python/monitoring-dashboards/latest/index.html) to create a dashboard based on the template.    2.1. Load the dashboard template<jupyter_code>dashboard_service_client = DashboardsServiceClient(credentials=creds)
parent = 'projects/{}'.format(PROJECT_ID)

dashboard_template_file = 'monitoring-template.json'
with open(dashboard_template_file) as f:
    dashboard_template = json.load(f)<jupyter_output><empty_output><jupyter_text>2.2. Create the dashboard in Cloud Monitoring<jupyter_code>dashboard_proto = Dashboard()
dashboard_proto = ParseDict(dashboard_template, dashboard_proto)
dashboard = dashboard_service_client.create_dashboard(parent, dashboard_proto)
for dashboard in dashboard_service_client.list_dashboards(parent):
    print('Dashboard name: {}, Dashboard ID: {}'.format(dashboard.display_name, dashboard.name))<jupyter_output><empty_output><jupyter_text>3. Deploying Locust to a GKE clusterBefore proceeding, you need access to a GKE cluster. The described deployment process can deploy Locust to any GKE cluster as long as there are enough compute resources to support your Locust configuration. The default configuration follows the Locust's best practices and requests one processor core and 4Gi of memory for the Locust master and one processor core and 2Gi of memory for each Locust worker. As you run your tests, it is important to monitor the the master and the workers for resource utilization and fine tune the allocated resources as required.The deployment process has been streamlined using [Kustomize](https://kustomize.io/). As described in the following steps, you can fine tune the baseline configuration by modifying the default `kustomization.yaml` and `patch.yaml` files in the `locust/manifests` folder.  3.1. Install Kustomize<jupyter_code>!curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  | bash
!sudo mv kustomize /usr/local/bin<jupyter_output><empty_output><jupyter_text>3.2. Set credentials to access your GKE clusterUse, the `gcloud` command to set credentials to your GKE cluster. Make sure to update the `cluster_name` and `cluster_zone` variables with values reflecting your environment.<jupyter_code>!gcloud container clusters get-credentials {GKE_CLUSTER_NAME} --zone {GKE_CLUSTER_ZONE}<jupyter_output><empty_output><jupyter_text>3.3. Build the Locust imageThe first step is to build a docker image that will be used to deploy Locust master and worker pods. The image is derived from the [baseline locust.io image](https://hub.docker.com/r/locustio/locust) and embeds the locustfile and the files's dependencies.<jupyter_code>image_uri = 'gcr.io/{}/locust'.format(PROJECT_ID)

!gcloud builds submit --tag {image_uri} locust/locust-image<jupyter_output><empty_output><jupyter_text>3.4. Update the manifestsBefore proceeding with deployment, you need to update the default manifests. The manifests are located in the `locust/manifests` folder. You will modify two files: `kustomization.yaml` and `patch.yaml`.1. **Set the name of the custom Locust image** - Update the `kustomization.yaml` file with a reference to the custom image your created in the previous step.  Update the `newName` field in the `images` section of the `kustomization.yaml` file.2. **Set the number of worker pods** - The default configuration deploys 32 worker pods. If you want to change it, modify the `count` field in the `replicas` section of the `kustomization.yaml` file. Make sure that your GKE cluster has as sufficient CPUs for the Locust workers.3. **Set the GCS bucket for the test configuration and data files** - When you start a test, the locustfile retrieves a test configuration and test data files from a GCS location. Modify the `configMapGenerator` section of the `kustomization.yaml`file to set the `LOCUST_TEST_BUCKET`, `LOCUST_TEST_CONFIG`, and `LOCUST_TEST_DATA` literals to the GCS bucket name, the test config file name, and the test data config file name respectively.4.  **Modify the node pool that hosts the Locust master and workers** - Update the name of the node pool in the `patch.yaml` file to which the Locust workload will be deployed. By default, the node pool name is `default-pool`. The name of the node pool is a value of the `values` field in the `matchExpressions` section.<jupyter_code>!cat locust/manifests/kustomization.yaml
!cat locust/manifests/patch.yaml<jupyter_output><empty_output><jupyter_text>3.5. Deploy LocustYou are now ready to deploy Locust.<jupyter_code>!kustomize build locust/manifests |kubectl apply -f -<jupyter_output><empty_output><jupyter_text>4. Configure a Locust testAt the start of each test, the locustfile script attempts to retrieve test data and a test configuration from a GCS location. Both the test data and the test configuration are formated as JSON. The test data is an array of JSON objects, where each object includes a list of instances and a model signature. If the array contains more than one object, Locust users will randomly pick a list of instances and an associated signature with each call to the `predict` method of the AI Platform Prediction endpoint.The test configuration is a JSON object with a project id, model name, model version, and a test id.  4.1. Prepare test dataIn this example we are using the  **ResNet101** model prepared and deployed in the `01-prepare-and-deply.ipynb` notebook. We will prepare the instances for the `serving_preprocess` signature of the model using a couple of JPEG images from the `test_images` folder.<jupyter_code>images = []
for image_name in os.listdir(IMAGES_FOLDER):
    with open(os.path.join(IMAGES_FOLDER, image_name), 'rb') as f:
        images.append(f.read())
single_instance = [{'b64': base64.b64encode(images[0]).decode('utf-8')}]
two_instances = [{'b64': base64.b64encode(image).decode('utf-8')} for image in images] 
test_data = [
        {
            'signature': 'serving_preprocess',
            'instances': single_instance
        },
        {
            'signature': 'serving_preprocess',
            'instances': two_instances
        }
    ]
test_data_local_file = 'test-payload.json'

with open (test_data_local_file, 'w') as f:
    json.dump(test_data, f)
    
!gsutil cp {test_data_local_file} {GCS_LOCUST_TEST_CONFIG_DIR}/<jupyter_output><empty_output><jupyter_text>4.2 Prepare test configMake sure to update the below mapping with the values representing your environment. The `test_id` is an arbitrary value that is used to match the custom log-based metrics records with a given test run. Use a different value anytime you start a test.<jupyter_code>test_id = 'test-{}'.format(datetime.datetime.now().strftime('%Y%m%d-%H%M%S'))

test_config = {
    'test_id': test_id,
    'project_id': PROJECT_ID,
    'model': MODEL_NAME,
    'version': MODEL_VERSION
}
test_config_local_file = 'test-config.json'

with open (test_config_local_file, 'w') as f:
    json.dump(test_config, f)

!gsutil cp {test_config_local_file} {GCS_LOCUST_TEST_CONFIG_DIR}/
!gsutil ls {GCS_LOCUST_TEST_CONFIG_DIR}<jupyter_output><empty_output><jupyter_text>5. Run the Locust load TestsLoad tests can be configured, started, monitored and stopped using using Locust's [web interface](https://docs.locust.io/en/stable/quickstart.htmllocust-s-web-interface). In our deployment, the web interface is exposed by an external load balancer. You can access the interface using the following URL:<jupyter_code>%%bash
IP_ADDRESS=$(kubectl get service locust-master | awk -v  col=4 'FNR==2{print $col}')
echo http://$IP_ADDRESS:8089<jupyter_output><empty_output><jupyter_text>You can try using the following parameter configurations:1. Number of total users to simulate: 1522. Hatch rate: 13. Host: http://ml.googleapis.com4. Number of users to increase by step: 85. Step duration: 1m   Cleaning up: remove the Locust deployment<jupyter_code>!kustomize build locust/manifests | kubectl delete -f -<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08166734666122671, 'fraction_numerical': 0.00931592547259622, 'mean_word_length': 5.002857142857143, 'pattern_counts': {'":': 0, '<': 73, '<?xml version=': 0, '>': 73, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '211887', 'license': 'Apache-2.0', 'n_tokens_mistral': 4441, 'n_tokens_neox': 4299, 'n_words': 1452, 'path': 'model_serving/caip-load-testing/02-perf-testing.ipynb', 'repo_name': 'jarokaz/mlops-on-gcp', 'size': '14706'}