<jupyter_start><jupyter_text>Huggingface Sagemaker-sdk - Deploy 🤗 Transformers for inference 1. [Introduction](Introduction)  2. [Deploy a trained Hugging Face Transformer model on to SageMaker for inference](Deploy-a-trained-Hugging-Face-Transformer-model-on-to-SageMaker-for-inference)      a. [Deploy the model directly after training](Deploy-the-model-directly-after-training)      b. [Deploy the model using `model_data`](Deploy-the-model-using-model_data)  3. [Deploy one of the 10 000+ Hugging Face Transformers to Amazon SageMaker for Inference](Deploy-one-of-the-10-000+-Hugging-Face-Transformers-to-Amazon-SageMaker-for-Inference)   Welcome to this getting started guide, we will use the new Hugging Face Inference DLCs and Amazon SageMaker Python SDK to deploy two transformer model for inference. In the first example we deploy a trained Hugging Face Transformer model on to SageMaker for inference.In the second example we directly deploy one of the 10 000+ Hugging Face Transformers from the [Hub](https://huggingface.co/models) to Amazon SageMaker for Inference.<  API - [SageMaker Hugging Face Inference Toolkit](https://github.com/aws/sagemaker-huggingface-inference-toolkit) Using the `transformers pipelines`, we designed an API, which makes it easy for you to benefit from all `pipelines` features. The API is oriented at the API of the [🤗  Accelerated Inference API](https://api-inference.huggingface.co/docs/python/html/detailed_parameters.html), meaning your inputs need to be defined in the `inputs` key and if you want additional supported `pipelines` parameters you can add them in the `parameters` key. Below you can find examples for requests. **text-classification request body**```python{	"inputs": "Camera - You are awarded a SiPix Digital Camera! call 09061221066 fromm landline. Delivery within 28 days."}```**question-answering request body**```python{	"inputs": {		"question": "What is used for inference?",		"context": "My Name is Philipp and I live in Nuremberg. This model is used with sagemaker for inference."	}}```**zero-shot classification request body**```python{	"inputs": "Hi, I recently bought a device from your company but it is not working as advertised and I would like to get reimbursed!",	"parameters": {		"candidate_labels": [			"refund",			"legal",			"faq"		]	}}```<jupyter_code>!pip install "sagemaker>=2.48.0" --upgrade<jupyter_output><empty_output><jupyter_text>Deploy a Hugging Face Transformer model from S3 to SageMaker for inference There are two ways on how you can deploy you SageMaker trained Hugging Face model from S3. You can either deploy it after your training is finished or you can deploy it later using the `model_data` pointing to you saved model on s3. Deploy the model directly after trainingIf you deploy you model directly after training you need to make sure that all required files are saved in your training script, including the Tokenizer and the Model. ```pythonfrom sagemaker.huggingface import HuggingFace pseudo code start  create HuggingFace estimator for running traininghuggingface_estimator = HuggingFace(....) starting the train job with our uploaded datasets as inputhuggingface_estimator.fit(...) pseudo code end  deploy model to SageMaker Inferencepredictor = huggingface_estimator.deploy(initial_instance_count=1, instance_type="ml.m5.xlarge") example request, you always need to define "inputs"data = {   "inputs": "Camera - You are awarded a SiPix Digital Camera! call 09061221066 fromm landline. Delivery within 28 days."} requestpredictor.predict(data)``` Deploy the model using `model_data`<jupyter_code>from sagemaker.huggingface import HuggingFaceModel
import sagemaker 

role = sagemaker.get_execution_role()

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
   model_data="s3://hf-sagemaker-inference/model.tar.gz",  # path to your trained sagemaker model
   role=role, # iam role with permissions to create an Endpoint
   transformers_version="4.6", # transformers version used
   pytorch_version="1.7", # pytorch version used
   py_version="py36", # python version of the DLC
)
# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
   initial_instance_count=1,
   instance_type="ml.m5.xlarge"
)
# example request, you always need to define "inputs"
data = {
   "inputs": "The new Hugging Face SageMaker DLC makes it super easy to deploy models in production. I love it!"
}

# request
predictor.predict(data)
# delete endpoint
predictor.delete_endpoint()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08360908890359585, 'fraction_numerical': 0.013677476285020957, 'mean_word_length': 5.314763231197771, 'pattern_counts': {'":': 9, '<': 10, '<?xml version=': 0, '>': 10, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '106914', 'license': 'Apache-2.0', 'n_tokens_mistral': 1377, 'n_tokens_neox': 1287, 'n_words': 512, 'path': 'sagemaker/10_deploy_model_from_s3/deploy_transformer_model_from_s3.ipynb', 'repo_name': 'hellosamstuart/notebooks', 'size': '4533'}