<jupyter_start><jupyter_text>Image Classification Model Deployment1. [Introduction](Introduction)4. [Deploy the Model for Realtime Inference](Deploy-the-Model-for-Realtime-Inference)    1. [Create endpoint configuration](Create-endpoint-configuration)     2. [Create endpoint](Create-endpoint)     3. [Perform inference](Perform-inference)     4. [Clean up](Clean-up)  1. IntroductionIn this notebook, I will deploy my image classification model for making real-time predictions.First, I need to set up the linkage and authentication to AWS services. There are three parts to this:* The roles used to give learning and hosting access to my data. This will automatically be obtained from the role used to start the notebook* The S3 bucket for training and model data* The Amazon sagemaker image classification docker image<jupyter_code>%%time
import time
from time import gmtime, strftime
import io
import numpy as np
import pickle
import s3fs
import boto3
import re
import sagemaker
from sagemaker import get_execution_role
from sagemaker.amazon.amazon_estimator import get_image_uri
from sagemaker.image_uris import retrieve

role = get_execution_role()

bucket = sagemaker.Session().default_bucket()

training_image = retrieve("image-classification", boto3.Session().region_name)

s3 = s3fs.S3FileSystem()
s3_client = boto3.client("s3")<jupyter_output>CPU times: user 1.38 s, sys: 185 ms, total: 1.56 s
Wall time: 10.5 s<jupyter_text>2. Deploy Model for Realtime InferenceI will now host the model with an endpoint and perform realtime inference.This section involves several steps,1. [Create endpoint configuration](CreateEndpointConfiguration) - Create a configuration defining an endpoint.1. [Create endpoint](CreateEndpoint) - Use the configuration to create an inference endpoint.1. [Perform inference](PerformInference) - Perform inference on some input data using the endpoint.1. [Clean up](CleanUp) - Delete the endpoint and model  a) Create Endpoint ConfigurationAt launch, I will support configuring REST endpoints in hosting with multiple models, e.g. for A/B testing purposes. In order to support this, we create an endpoint configuration, that describes the distribution of traffic across the models, whether split, shadowed, or sampled in some way.In addition, the endpoint configuration describes the instance type required for model deployment, and at launch will describe the autoscaling configuration.<jupyter_code># Model name
job_name_prefix = "PCAM-image-classification"
job_name = job_name_prefix + "-" + time.strftime("-%Y-%m-%d-%H-%M-%S", time.gmtime())
model_name = 'PCAM-image-classification-model-2021-10-04-17-55-20'

# sagemaker
sage = boto3.Session().client(service_name="sagemaker")
sagemaker = boto3.client("sagemaker")
# Endpoint configuration
timestamp = time.strftime("-%Y-%m-%d-%H-%M-%S", time.gmtime())
endpoint_config_name = job_name_prefix + "-epc-" + timestamp
endpoint_config_response = sage.create_endpoint_config(
    EndpointConfigName=endpoint_config_name,
    ProductionVariants=[
        {
            "InstanceType": "ml.m4.xlarge",
            "InitialInstanceCount": 1,
            "ModelName": model_name,
            "VariantName": "AllTraffic",
        }
    ],
)

print("Endpoint configuration name: {}".format(endpoint_config_name))
print("Endpoint configuration arn:  {}".format(endpoint_config_response["EndpointConfigArn"]))<jupyter_output>Endpoint configuration name: PCAM-image-classification-epc--2021-10-09-14-56-46
Endpoint configuration arn:  arn:aws:sagemaker:us-east-1:513033806411:endpoint-config/pcam-image-classification-epc--2021-10-09-14-56-46<jupyter_text>b) Create EndpointNext, I create the endpoint that serves up the model, through specifying the name and configuration defined above. The end result is an endpoint that can be validated and incorporated into production applications. This takes 9-11 minutes to complete.<jupyter_code># Create endpoint
timestamp = time.strftime("-%Y-%m-%d-%H-%M-%S", time.gmtime())
endpoint_name = job_name_prefix + "-ep-" + timestamp
print("Endpoint name: {}".format(endpoint_name))

endpoint_params = {
    "EndpointName": endpoint_name,
    "EndpointConfigName": endpoint_config_name,
}
endpoint_response = sagemaker.create_endpoint(**endpoint_params)
print("EndpointArn = {}".format(endpoint_response["EndpointArn"]))<jupyter_output>Endpoint name: PCAM-image-classification-ep--2021-10-09-14-59-55
EndpointArn = arn:aws:sagemaker:us-east-1:513033806411:endpoint/pcam-image-classification-ep--2021-10-09-14-59-55<jupyter_text>Now the endpoint can be created. It may take sometime to create the endpoint...<jupyter_code># get the status of the endpoint
response = sagemaker.describe_endpoint(EndpointName=endpoint_name)
status = response["EndpointStatus"]
print("EndpointStatus = {}".format(status))


# wait until the status has changed
sagemaker.get_waiter("endpoint_in_service").wait(EndpointName=endpoint_name)


# print the status of the endpoint
endpoint_response = sagemaker.describe_endpoint(EndpointName=endpoint_name)
status = endpoint_response["EndpointStatus"]
print("Endpoint creation ended with EndpointStatus = {}".format(status))

if status != "InService":
    raise Exception("Endpoint creation failed.")<jupyter_output>EndpointStatus = Creating
Endpoint creation ended with EndpointStatus = InService<jupyter_text>The following message indicates a functioning inference endpoint. You can confirm the endpoint configuration and status by navigating to the "Endpoints" tab in the AWS SageMaker console.> `Endpoint creation ended with EndpointStatus = InService`Next, I will create a runtime object from which I can invoke the endpoint.  c) Perform InferenceFinally, users can now validate the model for use. They can obtain the endpoint from the client library using the result from previous operations, and generate classifications from the trained model using that endpoint.<jupyter_code>runtime = boto3.Session().client(service_name="runtime.sagemaker")
# Download test image
s3_client = boto3.client("s3")

file_name = "test3.jpg"
object_name = "lymph-node-image-classification/images/color/test/test_3.jpg"
s3_client.download_file(bucket, object_name, file_name)

# test image
from IPython.display import Image

Image(file_name)<jupyter_output><empty_output><jupyter_text>Here I evaluate the image through the network for inference. The network outputs class probabilities and selects the class with the maximum probability as the final class output. This image is predicted as **positive** for metastatic cancer with a probability of **0.9999**.<jupyter_code>import json
import numpy as np

with open(file_name, "rb") as f:
    payload = f.read()
    payload = bytearray(payload)
response = runtime.invoke_endpoint(
    EndpointName=endpoint_name, ContentType="application/x-image", Body=payload
)
result = response["Body"].read()
# result will be in json format and convert it to ndarray
result = json.loads(result)
# the result will output the probabilities for all classes
# find the class with maximum probability and print the class index
index = np.argmax(result)
object_categories = ['Negative','Positive']
print("Result: label - " + object_categories[index] + ", probability - " + str(result[index]))<jupyter_output>Result: label - Positive, probability - 0.9999039173126221<jupyter_text>d) Clean upThat's it! The model has successfully been deployed for realtime inference! Now that I am finished with the endpoint, I can just delete it and the backing instances will be released.<jupyter_code>sage.delete_endpoint(EndpointName=endpoint_name)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09243697478991597, 'fraction_numerical': 0.021796218487394957, 'mean_word_length': 5.806970509383378, 'pattern_counts': {'":': 7, '<': 24, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '197957', 'license': 'MIT', 'n_tokens_mistral': 2231, 'n_tokens_neox': 2081, 'n_words': 702, 'path': 'model-deployment.ipynb', 'repo_name': 'jrbarclay37/PCam-Image-Classification', 'size': '7616'}