<jupyter_start><jupyter_text>SageMaker Serverless Inference NLP ExampleAmazon SageMaker Serverless Inference is a purpose-built inference option that makes it easy for customers to deploy and scale ML models. Serverless Inference is ideal for workloads which have idle periods between traffic spurts and can tolerate cold starts. Serverless endpoints also automatically launch compute resources and scale them in and out depending on traffic, eliminating the need to choose instance types or manage scaling policies.We will use a binary text classification example, which borrows from this code sample: https://github.com/huggingface/notebooks/tree/master/sagemaker/01_getting_started_pytorchNotebook Setting- SageMaker Classic Notebook Instance: ml.m5.xlarge Notebook Instance & conda_python3 Kernel- SageMaker Studio: Python 3 (Data Science)- Regions Available: SageMaker Serverless Inference is currently available in the following regions: US East (Northern Virginia), US East (Ohio), US West (Oregon), EU (Ireland), Asia Pacific (Tokyo) and Asia Pacific (Sydney)  Table of Contents- Setup- Model Training- Deployment    - Model Creation    - Endpoint Configuration (Adjust for Serverless)    - Serverless Endpoint Creation    - Endpoint Invocation- Cleanup  Development Environment and Permissions   SetupFor testing you need to properly configure your Notebook Role to have SageMaker Full Access. Let's start by installing preview wheels of the Python SDK, boto and aws cli  Installation_*Note:* we only install the required libraries from Hugging Face and AWS. You also need PyTorch or Tensorflow, if you haven´t it installed_<jupyter_code>%pip -q install torch boto3 sagemaker transformers datasets[s3] --user<jupyter_output><empty_output><jupyter_text>Permissions _If you are going to use Sagemaker in a local environment. You need access to an IAM Role with the required permissions for Sagemaker. You can find [here](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html) more about it._ Make sure Sagemaker version is >= 2.70.0<jupyter_code>import sagemaker
print(sagemaker.__version__)
sess = sagemaker.Session()
# sagemaker session bucket -> used for uploading data, models and logs
# sagemaker will automatically create this bucket if it not exists
sagemaker_session_bucket=None
if sagemaker_session_bucket is None and sess is not None:
    # set to default bucket if a bucket name is not given
    sagemaker_session_bucket = sess.default_bucket()

role = sagemaker.get_execution_role()
sess = sagemaker.Session(default_bucket=sagemaker_session_bucket)

print(f"sagemaker role arn: {role}")
print(f"sagemaker bucket: {sess.default_bucket()}")
print(f"sagemaker session region: {sess.boto_region_name}")<jupyter_output><empty_output><jupyter_text>PreprocessingWe are using the `datasets` library to download and preprocess the `imdb` dataset. After preprocessing, the dataset will be uploaded to our `sagemaker_session_bucket` to be used within our training job. The [imdb](http://ai.stanford.edu/~amaas/data/sentiment/) dataset consists of 25000 training and 25000 testing highly polar movie reviews.  Tokenization<jupyter_code>from datasets import load_dataset
from transformers import AutoTokenizer

# tokenizer used in preprocessing
tokenizer_name = 'distilbert-base-uncased'

# dataset used
dataset_name = 'imdb'

# s3 key prefix for the data
s3_prefix = 'samples/datasets/imdb'
# load dataset
dataset = load_dataset(dataset_name)

# download tokenizer
tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)

# tokenizer helper function
def tokenize(batch):
    return tokenizer(batch['text'], padding='max_length', truncation=True)

# load dataset
train_dataset, test_dataset = load_dataset('imdb', split=['train', 'test'])
test_dataset = test_dataset.shuffle().select(range(10000)) # smaller the size for test dataset to 10k 


# tokenize dataset
train_dataset = train_dataset.map(tokenize, batched=True)
test_dataset = test_dataset.map(tokenize, batched=True)

# set format for pytorch
train_dataset =  train_dataset.rename_column("label", "labels")
train_dataset.set_format('torch', columns=['input_ids', 'attention_mask', 'labels'])
test_dataset = test_dataset.rename_column("label", "labels")
test_dataset.set_format('torch', columns=['input_ids', 'attention_mask', 'labels'])<jupyter_output><empty_output><jupyter_text>Uploading data to `sagemaker_session_bucket`After we processed the `datasets` we are going to use the new `FileSystem` [integration](https://huggingface.co/docs/datasets/filesystems.html) to upload our dataset to S3.<jupyter_code>import botocore
from datasets.filesystems import S3FileSystem

s3 = S3FileSystem()  

# save train_dataset to s3
training_input_path = f's3://{sess.default_bucket()}/{s3_prefix}/train'
train_dataset.save_to_disk(training_input_path,fs=s3)

# save test_dataset to s3
test_input_path = f's3://{sess.default_bucket()}/{s3_prefix}/test'
test_dataset.save_to_disk(test_input_path,fs=s3)<jupyter_output><empty_output><jupyter_text>Fine-tuning & starting Sagemaker Training JobIn order to create a sagemaker training job we need an `HuggingFace` Estimator. The Estimator handles end-to-end Amazon SageMaker training and deployment tasks. In a Estimator we define, which fine-tuning script should be used as `entry_point`, which `instance_type` should be used, which `hyperparameters` are passed in .....```pythonhuggingface_estimator = HuggingFace(entry_point='train.py',                            source_dir='./scripts',                            base_job_name='huggingface-sdk-extension',                            instance_type='ml.p3.2xlarge',                            instance_count=1,                            transformers_version='4.4',                            pytorch_version='1.6',                            py_version='py36',                            role=role,                            hyperparameters = {'epochs': 1,                                               'train_batch_size': 32,                                               'model_name':'distilbert-base-uncased'                                                })```When we create a SageMaker training job, SageMaker takes care of starting and managing all the required ec2 instances for us with the `huggingface` container, uploads the provided fine-tuning script `train.py` and downloads the data from our `sagemaker_session_bucket` into the container at `/opt/ml/input/data`. Then, it starts the training job by running. ```python/opt/conda/bin/python train.py --epochs 1 --model_name distilbert-base-uncased --train_batch_size 32```The `hyperparameters` you define in the `HuggingFace` estimator are passed in as named arguments. Sagemaker is providing useful properties about the training environment through various environment variables, including the following:* `SM_MODEL_DIR`: A string that represents the path where the training job writes the model artifacts to. After training, artifacts in this directory are uploaded to S3 for model hosting.* `SM_NUM_GPUS`: An integer representing the number of GPUs available to the host.* `SM_CHANNEL_XXXX:` A string that represents the path to the directory that contains the input data for the specified channel. For example, if you specify two input channels in the HuggingFace estimator’s fit call, named `train` and `test`, the environment variables `SM_CHANNEL_TRAIN` and `SM_CHANNEL_TEST` are set.To run your training job locally you can define `instance_type='local'` or `instance_type='local_gpu'` for gpu usage. _Note: this does not working within SageMaker Studio_<jupyter_code>!pygmentize ./scripts/train.py<jupyter_output><empty_output><jupyter_text>Creating an Estimator and start a training job<jupyter_code>from sagemaker.huggingface import HuggingFace

# hyperparameters, which are passed into the training job
hyperparameters={'epochs': 1,
                 'train_batch_size': 32,
                 'model_name':'distilbert-base-uncased'
                 }
huggingface_estimator = HuggingFace(entry_point='train.py',
                            source_dir='./scripts',
                            instance_type='ml.p3.2xlarge',
                            instance_count=1,
                            role=role,
                            transformers_version='4.6',
                            pytorch_version='1.7',
                            py_version='py36',
                            hyperparameters = hyperparameters)
# starting the train job with our uploaded datasets as input
huggingface_estimator.fit({'train': training_input_path, 'test': test_input_path})<jupyter_output><empty_output><jupyter_text>Deployment After training the model, retrieve the model artifacts so that we can deploy the model to an endpoint.<jupyter_code># Retrieve model data from training job
model_artifacts = huggingface_estimator.model_data
model_artifacts<jupyter_output><empty_output><jupyter_text>Model CreationCreate a model by providing your model artifacts, the container image URI, environment variables for the container (if applicable), a model name, and the SageMaker IAM role.<jupyter_code># Setup clients
import boto3

client = boto3.client(service_name="sagemaker")
runtime = boto3.client(service_name="sagemaker-runtime")
region = boto3.session.Session().region_name

image_uri = sagemaker.image_uris.retrieve(
    framework='huggingface',
    base_framework_version='pytorch1.7',
    region=region,
    version='4.6',
    py_version='py36',
    instance_type='ml.m5.large',
    image_scope='inference'
)
image_uri
from time import gmtime, strftime

model_name = "hf-serverless" + strftime("%Y-%m-%d-%H-%M-%S", gmtime())
print("Model name: " + model_name)

create_model_response = client.create_model(
    ModelName=model_name,
    Containers=[
        {
            "Image": image_uri,
            "Mode": "SingleModel",
            "ModelDataUrl": model_artifacts,
        }
    ],
    ExecutionRoleArn=role,
)

print("Model Arn: " + create_model_response["ModelArn"])<jupyter_output><empty_output><jupyter_text>Endpoint Configuration CreationThis is where you can adjust the Serverless Configuration for your endpoint. The current max concurrent invocations for a single endpoint, known as MaxConcurrency, can be any value from 1 to 50, and MemorySize can be any of the following: 1024 MB, 2048 MB, 3072 MB, 4096 MB, 5120 MB, or 6144 MB.<jupyter_code>epc_name = "hf-serverless-epc" + strftime("%Y-%m-%d-%H-%M-%S", gmtime())

endpoint_config_response = client.create_endpoint_config(
    EndpointConfigName=epc_name,
    ProductionVariants=[
        {
            'VariantName': 'single-variant',
            'ModelName': model_name,
            'ServerlessConfig': {
                'MemorySizeInMB': 6144,
                'MaxConcurrency': 8,
            },
        },
    ],
)

print("Endpoint Configuration Arn: " + endpoint_config_response["EndpointConfigArn"])<jupyter_output><empty_output><jupyter_text>Serverless Endpoint CreationNow that we have an endpoint configuration, we can create a serverless endpoint and deploy our model to it. When creating the endpoint, provide the name of your endpoint configuration and a name for the new endpoint.<jupyter_code>endpoint_name = "hf-serverless-ep" + strftime("%Y-%m-%d-%H-%M-%S", gmtime())

create_endpoint_response = client.create_endpoint(
    EndpointName=endpoint_name,
    EndpointConfigName=epc_name,
)

print("Endpoint Arn: " + create_endpoint_response["EndpointArn"])<jupyter_output><empty_output><jupyter_text>Wait until the endpoint status is InService before invoking the endpoint.<jupyter_code># wait for endpoint to reach a terminal state (InService) using describe endpoint
import time

describe_endpoint_response = client.describe_endpoint(EndpointName=endpoint_name)

while describe_endpoint_response["EndpointStatus"] == "Creating":
    describe_endpoint_response = client.describe_endpoint(EndpointName=endpoint_name)
    print(describe_endpoint_response["EndpointStatus"])
    time.sleep(15)

describe_endpoint_response<jupyter_output><empty_output><jupyter_text>Endpoint InvocationInvoke the endpoint by sending a request to it. The following is a sample data point grabbed from the CSV file downloaded from the public Abalone dataset.<jupyter_code>import json
sentiment_input = {"inputs": "I love using the new Serverless Inference on SageMaker."}

response = runtime.invoke_endpoint(
    EndpointName=endpoint_name,
    Body=json.dumps(sentiment_input),
    ContentType="application/json",
)

print(response["Body"].read())<jupyter_output><empty_output><jupyter_text>Clean UpDelete any resources you created in this notebook that you no longer wish to use.<jupyter_code># client.delete_model(ModelName=model_name)
# client.delete_endpoint_config(EndpointConfigName=epc_name)
# client.delete_endpoint(EndpointName=endpoint_name)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08375378405650857, 'fraction_numerical': 0.009004113948614454, 'mean_word_length': 4.404362416107382, 'pattern_counts': {'":': 5, '<': 53, '<?xml version=': 0, '>': 55, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '68772', 'license': 'MIT', 'n_tokens_mistral': 3703, 'n_tokens_neox': 3567, 'n_words': 1115, 'path': '1_model_train_deploy.ipynb', 'repo_name': 'marshmellow77/nlp-serverless', 'size': '12883'}