<jupyter_start><jupyter_text>SageMaker Real-time Dynamic Batching Inference with Torchserve This notebook demonstrates the use of dynamic batching on SageMaker with [torchserve](https://github.com/pytorch/serve/) as a model server. It demonstrates the following1. Batch inference using DLC i.e. SageMaker's default backend container. This is done by using SageMaker python sdk in script-mode.2. Specifying inference parameters for torchserve using environment variables.3. Option to use a custom container with config file for torchserve baked-in the container. **Imports**<jupyter_code>import base64
import json
import numpy as np
import matplotlib.pyplot as plt
from PIL import Image
import os
import boto3, time, json
import sagemaker<jupyter_output><empty_output><jupyter_text>**Initiate session and retrieve region, account details**<jupyter_code>sm_sess = sagemaker.Session()
role = sagemaker.get_execution_role()
sess = boto3.Session()
region = sess.region_name
account = boto3.client("sts").get_caller_identity().get("Account")<jupyter_output><empty_output><jupyter_text>**Prepare model**<jupyter_code>bucket = sm_sess.default_bucket()
prefix = "ts-dynamic-batching"
model_file_name = "BERTSeqClassification"

!aws s3 cp s3://torchserve/tar_gz_files/BERTSeqClassification.tar.gz .
!aws s3 cp BERTSeqClassification.tar.gz s3://{bucket}/{prefix}/models/

f"s3://{bucket}/{prefix}/models/"
model_artifact = f"s3://{bucket}/{prefix}/models/{model_file_name}.tar.gz"
model_name = "hf-dynamic-torchserve-sagemaker"<jupyter_output><empty_output><jupyter_text>Use AWS Deep Learning Container<jupyter_code># We'll use a pytorch inference DLC image that ships with sagemaker-pytorch-inference-toolkit v2.0.6. This version includes support for Torchserve environment variables used below.
image_uri = sagemaker.image_uris.retrieve(
    framework="pytorch",
    region=region,
    py_version="py38",
    image_scope="inference",
    version="1.9",
    instance_type="ml.c5.9xlarge",
)
image_uri<jupyter_output><empty_output><jupyter_text>Create SageMaker model, deploy and predict<jupyter_code>from sagemaker.pytorch.model import PyTorchModel

env_variables_dict = {
    "SAGEMAKER_TS_BATCH_SIZE": "3",
    "SAGEMAKER_TS_MAX_BATCH_DELAY": "100000",
    "SAGEMAKER_TS_MIN_WORKERS": "1",
    "SAGEMAKER_TS_MAX_WORKERS": "1",
}

pytorch_model = PyTorchModel(
    model_data=model_artifact,
    role=role,
    image_uri=image_uri,
    source_dir="code",
    framework_version="1.9",
    entry_point="inference.py",
    env=env_variables_dict,
)
# Change the instance type as necessary, or use 'local' for executing in Sagemaker local mode
instance_type = "ml.c5.18xlarge"

predictor = pytorch_model.deploy(
    initial_instance_count=1,
    instance_type=instance_type,
    serializer=sagemaker.serializers.JSONSerializer(),
    deserializer=sagemaker.deserializers.BytesDeserializer(),
)<jupyter_output><empty_output><jupyter_text>Predictions  By spawning a pool of 3 processes we're able to simulate requests from multiple clients and verify inference results<jupyter_code>import multiprocessing


def invoke(endpoint_name):
    predictor = sagemaker.predictor.Predictor(
        endpoint_name,
        sm_sess,
        serializer=sagemaker.serializers.JSONSerializer(),
        deserializer=sagemaker.deserializers.BytesDeserializer(),
    )
    return predictor.predict(
        "{Bloomberg has decided to publish a new report on global economic situation.}"
    )


endpoint_name = predictor.endpoint_name
pool = multiprocessing.Pool(3)
results = pool.map(invoke, 3 * [endpoint_name])
pool.close()
pool.join()
print(results)
predictor.delete_endpoint(predictor.endpoint_name)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09940572663425176, 'fraction_numerical': 0.011075094543490005, 'mean_word_length': 5.844731977818854, 'pattern_counts': {'":': 4, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '481617', 'license': 'Apache-2.0', 'n_tokens_mistral': 1241, 'n_tokens_neox': 1212, 'n_words': 266, 'path': 'sagemaker-python-sdk/pytorch_batch_inference/sagemaker_batch_inference_torchserve.ipynb', 'repo_name': 'vllyakho/amazon-sagemaker-examples', 'size': '3702'}