<jupyter_start><jupyter_text>SageMaker Inference with mmdetection3DThis notebook will use the model you trained earlier in the workshop to perform inference using [mmdetection3D](https://github.com/open-mmlab/mmdetection3d).Point cloud data can take up a lot of space. Because pointcloud payloads may be quite large, we will use SageMaker's [Asynchronous Inference](https://docs.aws.amazon.com/sagemaker/latest/dg/async-inference.html) capability. The notebook will follow these basic steps:* Define a SageMaker model using the results from our training process* Create an async inference endpoint* Test the endpoint Async InferenceSageMaker real-time inference endpoints typically act in a synchronous mode, just like any RESTful API.  That poses a challenge for cases where the input or output are too large for a normal REST payload.  SageMaker async inference endpoints solve that problem.![Async Inference](display_images/async-inf.png)_Image copied from a [SageMaker example notebook](https://github.com/aws/amazon-sagemaker-examples/blob/master/async-inference/Async-Inference-Walkthrough.ipynb)_When a request comes in to an async inference endpoint, the request goes into an internal queue, and references input data stored in an S3 bucket.  When the inference code runs, it gets the input data from S3 and gets the inference.  SageMaker stores the result in another S3 object, which we can download later.See this [example notebook](https://github.com/aws/amazon-sagemaker-examples/blob/master/async-inference/Async-Inference-Walkthrough.ipynb) for a more detailed walkthrough of async inference endpoints. Inference I/O formatSince we are providing our own inference code, we can determine the input and output formats.  The input to the inference endpoint is a point cloud from A2D2.  The response is a [pickled](https://docs.python.org/3/library/pickle.html) version of the mmdetect response object.The model artifact has to contain a configuration file and a model checkpoint.  The configuration file can point to other items in the `mmdetection3d` folder, but it has to contain absolute paths.  In the image,`mmdetection3d` is installed directly under the root.We have the mmdetect configuration file and model checkpoint prepared from the previous notebook. Inference codeOur inference container runs a [Flask](https://flask.palletsprojects.com/en/2.0.x/) server to respond to inference requests.  You can review the code in the `container_inference` directory.  The files `serve`, `nginx.conf`, and `wsgi.py` are boilerplate for the Flask server.  The interesting code is in `predictor.py`.<jupyter_code>!pygmentize -l python container_inference/mm3d/predictor.py<jupyter_output>[37m# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.[39;49;00m

[34mfrom[39;49;00m [04m[36m__future__[39;49;00m [34mimport[39;49;00m print_function

[34mimport[39;49;00m [04m[36mos[39;49;00m
[34mimport[39;49;00m [04m[36mio[39;49;00m
[34mimport[39;49;00m [04m[36mjson[39;49;00m
[34mimport[39;49;00m [04m[36mtempfile[39;49;00m
[34mimport[39;49;00m [04m[36mpickle[39;49;00m

[34mimport[39;49;00m [04m[36mflask[39;49;00m

[34mimport[39;49;00m [04m[36mtorch[39;49;00m
[34mfrom[39;49;00m [04m[36mglob[39;49;00m [34mimport[39;49;00m glob

[34mfrom[39;49;00m [04m[36mmmdet3d[39;49;00m[04m[36m.[39;49;00m[04m[36mapis[39;49;00m [34mimport[39;49;00m init_model, inference_detector

prefix = [33m'[39;49;00m[33m/opt/ml/[39;49;00m[33m'[39;49;00m
model_path = os.path.join(prefix, [33m'[39;49;00m[33mmodel[39;49;00m[33m'[39;49;00m)

[37m# A singleton for holding the model. This simply loads the model and holds it.[[...]<jupyter_text>You'll notice that the `predictor.py` code handles two Flask methods.  The first, `ping`, is just a health check that lets SageMaker know that the endpoint is ready to serve.  In that method, we see if we can load the model successfully.The second method, `transformation`, actually returns an inference.    Install pytransform3dWe will install pytransform3d that we will use later to visualize our inference results.<jupyter_code>!pip install pytransform3d<jupyter_output><empty_output><jupyter_text>Upload model artifactWhen you train a model in SageMaker it packages the results in a compressed `model.tar.gz` file. This file contains the trained model weights and any model configuration files. SageMaker inference requires your model to be in this format, but since we trained our model in SageMaker it's already in the correct format. Let's upload this file to S3 so our endpoint can retrieve it later.<jupyter_code># Use the default notebook execution role
from sagemaker import get_execution_role

role = get_execution_role()
from sagemaker.s3 import S3Uploader
import sagemaker
import boto3

# Upload model artifact to S3
file_key = 'model.tar.gz'
sm_session = sagemaker.session.Session()
bucket = sm_session.default_bucket()
prefix = "mm3d"
b3sess = boto3.Session()
region = b3sess.region_name
model_artifact = S3Uploader.upload(file_key, f's3://{bucket}/{prefix}/model')
print(model_artifact)<jupyter_output><empty_output><jupyter_text>Deployment containerIf running this in an AWS instructor led workshop, your container may have already been built ahead of time, but if running this on your own, you'll need to build your deployment container.SageMaker inference supplies a variety of different built in inference containers, since `mmdetection3d` is a library with its own set of complex dependencies, we are going to use a custom hosting container. This process will take ~10 minutes.<jupyter_code>IMAGE_NAME = 'mm3dinf'

# if in AIM414 your container will already be built for you!
# logging to Sagemaker ECR with Deep Learning Containers
# !aws ecr get-login-password --region {region} | docker login --username AWS --password-stdin <PASSWORD>.dkr.ecr.{region}.amazonaws.com
# !bash ./build_and_push.sh {region} {IMAGE_NAME} latest container_inference 
import time
account = boto3.client('sts').get_caller_identity()['Account']

container = f'{account}.dkr.ecr.us-east-1.amazonaws.com/{IMAGE_NAME}'
model_name = f'sagemaker-mm3d-{int(time.time())}'
print(container)
print(model_name)<jupyter_output>427894311213.dkr.ecr.us-east-1.amazonaws.com/mm3dinf
sagemaker-mm3d-1637716167<jupyter_text>Create SageMaker modelBefore we deploy our endpoint, we need to create a SageMaker model. A SageMaker model is different from the model we trained in the previous notebook in that it contains the information necessary to create the environment around the model so that it can be deployed as an endpoint when paired with an endpoint configuration, which will tell SageMaker what kind of instance to deploy the model to.<jupyter_code># Define SageMaker model using our container and model artifact

sm_client = boto3.client("sagemaker")

create_model_response = sm_client.create_model(
    ModelName = model_name,
    ExecutionRoleArn = role,
    PrimaryContainer = {
        'Image': container,
        'ModelDataUrl': model_artifact,
        'Environment': {
            'TS_MAX_REQUEST_SIZE': '100000000', 
            'TS_MAX_RESPONSE_SIZE': '100000000',
            'TS_DEFAULT_RESPONSE_TIMEOUT': '1000'
        }
    },    
)
# Confirm that the model was created
create_model_response<jupyter_output><empty_output><jupyter_text>Create inference endpointIn this section, we'll create an async inference endpoint. We take the model object we created previously and pair it with an endpoint configuration that tell SageMaker how many and what type of instances to deploy the model to. In this case we also use the endpoint configuration to tell SageMaker we want to make this endpoint an asynchronous endpoint.<jupyter_code># Define an endpoint that uses one ml.g4dn.2xlarge instance and uses async inference
from time import strftime,gmtime
endpoint_config_name = f"MM3DAsyncEndpointConfig-{strftime('%Y-%m-%d-%H-%M-%S', gmtime())}"
create_endpoint_config_response = sm_client.create_endpoint_config(
    EndpointConfigName=endpoint_config_name,
    ProductionVariants=[
        {
            "VariantName": "variant1",
            "ModelName": model_name,
            "InstanceType": "ml.g4dn.2xlarge",
            "InitialInstanceCount": 1
        }
    ],
    AsyncInferenceConfig={
        "OutputConfig": {
            "S3OutputPath": f"s3://{bucket}/{prefix}/output"
        },
        "ClientConfig": {
            "MaxConcurrentInvocationsPerInstance": 1
        }
    }
)
print(f"Created EndpointConfig: {create_endpoint_config_response['EndpointConfigArn']}")<jupyter_output>Created EndpointConfig: arn:aws:sagemaker:us-east-1:427894311213:endpoint-config/mm3dasyncendpointconfig-2021-11-24-01-09-32<jupyter_text>Now that we have created our SageMaker model object and our endpoint configuration, we can deploy our endpoint. The deployment process takes 5-10 minutes.<jupyter_code># Deploy the endpoint
endpoint_name = f"sm-{strftime('%Y-%m-%d-%H-%M-%S', gmtime())}"
create_endpoint_response = sm_client.create_endpoint(EndpointName=endpoint_name, EndpointConfigName=endpoint_config_name)
print(f"Creating Endpoint: {create_endpoint_response['EndpointArn']}")<jupyter_output>Creating Endpoint: arn:aws:sagemaker:us-east-1:427894311213:endpoint/sm-2021-11-24-01-09-42<jupyter_text>Running the following cell will launch a waiter that will alert us when our endpoint has been successfully deployed.<jupyter_code># Wait for the endpoint to enter service
waiter = boto3.client('sagemaker').get_waiter('endpoint_in_service')
print("Waiting for endpoint to create...")
waiter.wait(EndpointName=endpoint_name)
resp = sm_client.describe_endpoint(EndpointName=endpoint_name)
print(f"Endpoint Status: {resp['EndpointStatus']}")<jupyter_output>Waiting for endpoint to create...
Endpoint Status: InService<jupyter_text>Test the endpointNext we'll upload an A2D2 point cloud file and test the inference endpoint.  Since we're using async inference, the call to the endpoint doesn't return the inference instantaneously.  Rather, it returns a pointer to the S3 output location.  We can check for the presence of the results there.<jupyter_code># Helper method for uploading to S3
def upload_file(input_location, prefix):
    prefix = f"{prefix}/input"
    return sm_session.upload_data(
        input_location, 
        bucket=sm_session.default_bucket(),
        key_prefix=prefix)<jupyter_output><empty_output><jupyter_text>First we upload our point cloud file to a location in S3. Let's gather the paths to our scene IDs and pick a scene and a frame.<jupyter_code>from glob import glob

scene_ids = sorted(glob('../fsx/a2d2/camera_lidar_semantic_bboxes/2018*'))
scene_ids
dataset_root_path = "../fsx/a2d2/"
scene_id = "20180807_145028"
file_scene_prefix = scene_id.replace("_", "")
frame_id = "000000091"

input_1_location = f"../fsx/a2d2/camera_lidar_semantic_bboxes/{scene_id}/lidar/cam_front_center/{file_scene_prefix}_lidar_frontcenter_{frame_id}.bin"
input_1_s3_location = upload_file(input_1_location, prefix)
print(input_1_s3_location)
sm_runtime = boto3.client("sagemaker-runtime")
# Invoke endpoint using boto3 API
response = sm_runtime.invoke_endpoint_async(
    EndpointName=endpoint_name, 
    InputLocation=input_1_s3_location)
output_location = response['OutputLocation']
print(f"OutputLocation: {output_location}")
# Parse the output S3 location from the response
from urllib.parse import urlparse
parse_object = urlparse(output_location)
parse_object.netloc
parse_object.path[1:]
# Download the output.  
s3_client = boto3.client('s3')
s3_client.download_file(parse_object.netloc, parse_object.path[1:], 'test.pkl')
# The output is a pickled Python object.  Convert it back to a dictionary.
import pickle
with open('test.pkl', 'rb') as test_f:
    test_out = pickle.load(test_f)
# View the output
test_out
import cv2
import numpy as np
import matplotlib.pyplot as plt
from pathlib import Path
from pytransform3d import plot_utils
from pytransform3d import rotations as pr
from pytransform3d import transformations as pt

from a2d2_helpers import undistort_image, generate_color_map


file_scene_prefix = scene_id.replace("_", "")
scene_path = Path(dataset_root_path) / "camera_lidar_semantic_bboxes" / scene_id
frame_image_path = scene_path / "camera" / "cam_front_center" / f"{file_scene_prefix}_camera_frontcenter_{frame_id}.png"
frame_lidar_path = scene_path / "lidar" / "cam_front_center" / f"{file_scene_prefix}_lidar_frontcenter_{frame_id}.npz"
frame_label_path = scene_path / "label3D" / "cam_front_center" / f"{file_scene_prefix}_label3D_frontcenter_{frame_id}.json"

# Load our frame point cloud from the compressed NPZ format, grabbing the points and reflectance corresponding
# to those points. We'll use the reflectance to color our points in the 3D plot.
lidar_frame = np.load(str(frame_lidar_path))
points = lidar_frame["points"]
reflectance = lidar_frame["reflectance"]


fig = plt.figure(figsize=(20,10), dpi=200)
ax_3d = fig.add_subplot(1, 2, 1, projection='3d')

# Plot our point cloud.
ax_3d.scatter(points[:,0], points[:,1], points[:,2], s=0.1, c=reflectance, marker=",")

inference_score_pairs = list(zip(test_out["result"]["boxes_3d"], test_out["result"]["scores_3d"]))
inference_score_pairs = sorted(inference_score_pairs, key=lambda inf_score_pair: -inf_score_pair[1])

for inference, score in inference_score_pairs[:5]:
    x, y, z, l, w, h, yaw = inference
    # See https://mmdetection3d.readthedocs.io/en/latest/api.html for details on mmdetection output format.
    qw, qx, qy, qz = pr.quaternion_from_matrix(pr.matrix_from_axis_angle([0, 0, 1, np.pi / 2 - yaw]))
    A2B = pt.transform_from_pq([x, y, z, qw, qx, qy, qz])
    size = [w, l, h]

    plot_utils.plot_box(ax_3d, size=size, A2B=A2B, alpha=0.4)
    plot_utils.Frame(A2B).add_frame(ax_3d)

# Load our frame image from the file system.
image_frame = cv2.imread(str(frame_image_path))
image_frame = cv2.cvtColor(image_frame, cv2.COLOR_BGR2RGB)


# Show image side by side.
ax_2d = fig.add_subplot(1, 2, 2)
ax_2d.imshow(image_frame)
ax_2d.set_title("Image with projected labels overlayed")


limit_m = 10
translation = [25, 0, 0]
ax_3d.set_xlim((-limit_m + translation[0], limit_m + translation[0]))
ax_3d.set_ylim((-limit_m + translation[1], limit_m + translation[1]))
ax_3d.set_zlim((-limit_m + translation[2], limit_m + translation[2]))
ax_3d.set_xlabel("X")
ax_3d.set_ylabel("Y")
ax_3d.set_zlabel("Z")
ax_3d.view_init(5, 180)<jupyter_output><empty_output><jupyter_text>Local mode testingIf you want to test inference in local mode, you can use this section of the notebook.  You'll need to edit the location of the input point cloud file.  Also note that we specify the SHA of the current version of the container image, so that the local mode endpoint knows when we have a new image to test. In our case we are just going to use the latest version of the container.<jupyter_code># EDIT CONTAINER IMAGE URL AND SHA IN THIS CELL

container = f'{account}.dkr.ecr.us-east-1.amazonaws.com/mm3dinf'
container_sha = ''
# you can optionally add a SHA specification to the container to use a specific version, we will just use the latest version
# simply replace f"{container}" with f"{container}@{container_sha}"
s_model = sagemaker.model.Model(f"{container}", model_data=model_artifact, role=role, predictor_cls=sagemaker.predictor.Predictor)
import torch 
instance_type = "local_gpu" if torch.cuda.is_available() else "local"

# Deploy a new local endpoint
predictor = s_model.deploy(
        initial_instance_count=1,
        instance_type=instance_type,
        
    )
input_1_location
# this is using the point cloud location we gave to our asynchronous endpoint above, you can change it to test different files

with open(input_1_location, 'rb') as in_file:
    d = in_file.read()
# Get prediction locally
r = predictor.predict(d)
# Deserialize the output
import pickle
rr = pickle.loads(r)
# let's check our results
rr<jupyter_output><empty_output><jupyter_text>CleanupMake sure to delete unused endpoints, especially if you are running this in your own account!<jupyter_code>response = sm_client.delete_endpoint(
    EndpointName=endpoint_name
)
response<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09700894233734196, 'fraction_numerical': 0.03546099290780142, 'mean_word_length': 5.5999185999186, 'pattern_counts': {'":': 8, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 3, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '331576', 'license': 'ECL-2.0', 'n_tokens_mistral': 5574, 'n_tokens_neox': 5122, 'n_words': 1551, 'path': '3_inference_byom.ipynb', 'repo_name': 'aws-samples/end-2-end-3d-ml', 'size': '16217'}