<jupyter_start><jupyter_text>BYO Serving Container on SageMakerIn this notebook, we will develop SageMaker-compatible container for inference. We will use latest TensorFlow container as a base image and use AWS Multi-Model Server ("MMS") as a model server. Please note that MMS is one of several ML model serving options. Problem OverviewWe will use pre-trained [VGG16 model](https://arxiv.org/pdf/1409.1556.pdf) to classify content of the images into 1000 categories. The model is trained on ImageNet dataset.We will use Keras Deep Learning library which is now a part of TensorFlow code base. Hence, we choose choose latest TensorFlow container as a base.  Developing Serving ContainerWhen deploying serving container to endpoint SageMaker runs `docker run  serve` command. To comply with this requirement it's reccommended to use exec format of ENTRYPOINT instruction in your Dockerfile.Let's review our BYO Dockerfile:- we use latest tensorflow-devel container as base.- we install general and SageMaker specific dependencies.- we copy our model serving scripts to container.- we specify ENTRYPOINT and CMD instructions to comply with SageMaker requirements.<jupyter_code>! pygmentize -O linenos=1 -l docker 3_sources/Dockerfile.inference<jupyter_output><empty_output><jupyter_text>Developing Model Serving ScriptsInference scripts in case of BYO container are specific to chosen model server. In our case we are using AWS MMS server and developed scripts according to it's requirements. You find more details here: https://github.com/awslabs/multi-model-server/blob/master/docs/custom_service.mdIn this example we don't intend to cover MMS and development of inference scripts in details. However, it's worth highlighting some key script aspects:- `dockerd_entrypoint.py` is an excuitable which starts MMS server when `serve` argument is passed to it.- `model_handler.py` implements model loading and model serving logics. Note, that method `handle()` checks if model is already loaded into memory. If it's not, it will load model into memory once and then proceed to handling serving request which includes:    - deserializing request payload.    - running predictions.    - serializing predictions.<jupyter_code>! pygmentize 3_sources/src/dockerd_entrypoint.py
! pygmentize 3_sources/src/model_handler.py<jupyter_output><empty_output><jupyter_text>Building BYO ContainerOnce we have Dockerfile and inference scripts are ready, we can proceed and build container. We start by importing SageMaker utilities and providing configuration settings for our container and SageMaker model.<jupyter_code>import sagemaker, boto3
from sagemaker import get_execution_role

session = sagemaker.Session()
role = get_execution_role()
account = boto3.client('sts').get_caller_identity().get('Account')
region = session.boto_region_name

# Configuration settings
model_name="vgg16-model"
endpoint_name= model_name+"-mms-endpoint"
tag = "v1"
image_uri = f"{account}.dkr.ecr.{region}.amazonaws.com/{model_name}:{tag}"<jupyter_output><empty_output><jupyter_text>Now, we need to authenticate in our private ECR before we can push there BYO container image.<jupyter_code># loging to your private ECR
!aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin {account}.dkr.ecr.{region}.amazonaws.com<jupyter_output><empty_output><jupyter_text>After that we are ready to build BYO container and push it to ECR.<jupyter_code>!./build_and_push.sh {model_name} {tag} 3_sources/Dockerfile.inference<jupyter_output><empty_output><jupyter_text>Deploying SageMaker EndpointWe use generic `Model` object to configure SageMaker model and endpoint which allows us to use BYO container image. Note, that since we download model from public model zoo, we don't need to provide `model_data`.<jupyter_code>from sagemaker import Model

mms_model = Model(
    image_uri=image_uri,
    model_data=None,
    role=role,
    name=model_name,
    sagemaker_session=session
)

mms_model.deploy(
    initial_instance_count=1,
    instance_type="ml.m5.xlarge", 
    endpoint_name=endpoint_name
)<jupyter_output><empty_output><jupyter_text>Test SageMaker EndpointTo test the endpoint we will use a sample image. Feel free to pick several other images of your choice (make sure they have object belonging to one of 1000 categories from ImageNet).<jupyter_code>TEST_IMAGE = "sample_image.jpg"
! wget -O {TEST_IMAGE} https://farm1.static.flickr.com/56/152004091_5bfbc69bb3.jpg<jupyter_output><empty_output><jupyter_text>VGG16 model expects an image of size 224x224 pixels.<jupyter_code>%matplotlib inline
import cv2
import numpy as np
from matplotlib import pyplot as plt

def resize_image(filename):
    img = cv2.imread('152004091_5bfbc69bb3.jpg')
    resized_img = cv2.resize(img, dsize=(224, 224), interpolation=cv2.INTER_CUBIC)
    resized_filename = "resized_"+TEST_IMAGE

    cv2.imwrite(resized_filename, resized_img)

    plt.imshow(cv2.imread(resized_filename))
    plt.show()
    
    return resized_filename

resized_test_image = resize_image(TEST_IMAGE)<jupyter_output><empty_output><jupyter_text>To test the endpoint, we will use `boto3.sagemaker-runtime` client which allows to construct HTTP request and send it to defined SageMaker endpoint.<jupyter_code>import boto3

client = boto3.client('sagemaker-runtime')
accept_type = "application/json"
content_type = 'image/jpeg'
headers = {'content-type': content_type}
payload = open(resized_test_image, 'rb')

response = client.invoke_endpoint(
    EndpointName=endpoint_name,
    Body=payload,
    ContentType=content_type,
    Accept = accept_type
)


most_likely_label = response['Body'].read()

print(most_likely_label)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07518401682439538, 'fraction_numerical': 0.01454609183315808, 'mean_word_length': 5.745862884160757, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '617256', 'license': 'MIT', 'n_tokens_mistral': 1767, 'n_tokens_neox': 1671, 'n_words': 570, 'path': 'chapter2/3_BYO_Serving_Container.ipynb', 'repo_name': 'PacktPublishing/Accelerate-Deep-Learning-Workloads-with-Amazon-SageMaker', 'size': '5706'}