<jupyter_start><jupyter_text>Precondition: the model is uploaded in s3 by training job or manually<jupyter_code>model_data="s3://sagemaker-us-east-1-420737321821/sagemaker/ivy-demo-pytorch-bert/pytorch-training-2021-11-29-14-48-31-535/output/model.tar.gz"
import os
import numpy as np
import pandas as pd
import sagemaker

sagemaker_session = sagemaker.Session()

bucket = sagemaker_session.default_bucket()
prefix = "sagemaker/ivy-demo-pytorch-bert"

role = sagemaker.get_execution_role()
print(bucket)
print(role)<jupyter_output>sagemaker-us-east-1-420737321821
arn:aws:iam::420737321821:role/CUSPFE-SageMaker-ML-Team-Execution-Role-Test<jupyter_text>1. Create Pytorch endpoint with predefined the pytorch framework- it will create the inference model and endpoint configure automatically- it will use the awe prebuilt pytorch framework, we don't need to build our own docker imag, but need to prepare the entrypoint script<jupyter_code>from sagemaker.pytorch import PyTorchModel

# instance_type = 'ml.m5.large'
# accelerator_type = 'ml.eia2.xlarge'
instance_type = 'ml.t2.medium'
import time

endpoint_name = 'grammar-classification-{}-ep'.format(time.time()).replace('.', '').replace('_', '')
print(endpoint_name)
pytorch = PyTorchModel(
    model_data=model_data,
    role=role,
    entry_point='train_deploy.py',
    source_dir='code',
    framework_version='1.3.1',
    py_version='py3',
    sagemaker_session=sagemaker_session
)

# Function will exit before endpoint is finished creating
predictor = pytorch.deploy(
    initial_instance_count=1,
    instance_type=instance_type,
#     accelerator_type=accelerator_type,
    endpoint_name=endpoint_name,
    wait=True,
)<jupyter_output>grammar-classification-16383498089475572-ep
----------!<jupyter_text>2. Test the endpointthe code can run outside sagemaker, like lambda function<jupyter_code>import boto3
sagemaker_runtime_client = boto3.Session().client(service_name='runtime.sagemaker')
import json
def predict_grammer_text(text):
    response = sagemaker_runtime_client.invoke_endpoint(
        EndpointName=endpoint_name, 
        ContentType='application/json', 
        Body=json.dumps(text)
    )
    result = response['Body'].read()
    result = json.loads(result)
    print('Probabilities for all classes: ', result)
    predicted_class = np.argmax(result)
    if predicted_class == 0:
        print('Grammer incorrect!')
    else:
        print('Grammer correct.')
    return predicted_class
s = "I am a girl come from Chinese."
x = predict_grammer_text(s)
print(x)
s = "I am a girl from China."
x = predict_grammer_text(s)
print(x)<jupyter_output>Probabilities for all classes:  [[-0.879428505897522, 1.862220287322998]]
Grammer correct.
1<jupyter_text>3. Clean the endpointPlease remember to clean the endpoint if it is just for practicethe endpoint will create ec2 instance in the backend to serve the predict request. if we didn't clean the endpoint, the ec2 instance will keep running in the backend which will generate cost.<jupyter_code>print(predictor.endpoint_name)
resp = sagemaker_session.delete_endpoint(endpoint_name=predictor.endpoint_name)
print(type(resp), resp)

resp = sagemaker_session.delete_endpoint_config(endpoint_config_name=predictor.endpoint_name)
print(type(resp), resp)
print(pytorch_model.name)
resp = sagemaker_session.delete_model(model_name=pytorch_model.name)
print(type(resp), resp)<jupyter_output><class 'numpy.int64'> None
{'fraction_non_alphanumeric': 0.1026538349372995, 'fraction_numerical': 0.03645377661125693, 'mean_word_length': 5.558317399617591, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 14, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '170727', 'license': 'MIT-0', 'n_tokens_mistral': 1207, 'n_tokens_neox': 1130, 'n_words': 252, 'path': 'grammar-classification-deploy-prebuilt-dockerimg.ipynb', 'repo_name': 'songm28/amazon-sagemaker-bert-pytorch', 'size': '3429'}