<jupyter_start><jupyter_text>Question AnsweringIn this notebook, we'll deploy and use a question answering modelfrom the [transformers](https://huggingface.co/transformers/) librarywhich uses PyTorch.Question Answering is useful when you want to query a large amount oftext for specific information. Maybe you're interested in extracting thedate a certain event happened. You can construct a question (or query) innatural language to retrive this information: e.g. 'When did Company Xrelease Product Y?". Similar to extractive summarization we saw in thelast notebook, Question Answering will return a verbatim slice of thetext as the answer. It won't generate new words to answer the question.**Note**: When running this notebook on SageMaker Studio, you should makesure the 'SageMaker JumpStart PyTorch 1.0' image/kernel is used. Whenrunning this notebook on SageMaker Notebook Instance, you should makesure the 'sagemaker-soln' kernel is used. We start by importing a variety of packages that will be used throughoutthe notebook. One of the most important packages is the Amazon SageMakerPython SDK (i.e. `import sagemaker`). We also import modules from our owncustom (and editable) package that can be found at `../package`.<jupyter_code>import boto3
import sagemaker
from sagemaker.pytorch import PyTorchModel
from sagemaker.local import LocalSession
from sagemaker.predictor import json_serializer, json_deserializer
import sys

sys.path.insert(0, '../package')
from package import config, utils<jupyter_output><empty_output><jupyter_text>Up next, we define the current folder and create a SageMaker client (from`boto3`). We can use the SageMaker client to call SageMaker APIsdirectly, as an alternative to using the Amazon SageMaker SDK. We'll useit at the end of the notebook to delete certain resources that arecreated in this notebook.<jupyter_code>current_folder = utils.get_current_folder(globals())
sagemaker_client = boto3.client('sagemaker')<jupyter_output><empty_output><jupyter_text>Our question answering system needs a machine learning model. In thissection, we'll deploy a model to an Amazon SageMaker Endpoint and theninvoke the endpoint from the notebook. We'll use a pre-trained model fromthe [transformers](https://huggingface.co/transformers/) library insteadof training a model from scratch, specifically the BERT Large model thathas been pre-trained on the SQuAD dataset.We'll use the unique solution prefix to name the model and endpoint.<jupyter_code>model_name = "{}-question-answering".format(config.SOLUTION_PREFIX)<jupyter_output><empty_output><jupyter_text>Up next, we need to define the Amazon SageMaker Model which referencesthe source code and the specifies which container to use. Our pre-trainedmodel is from the transformers library which uses PyTorch. As a result,we should use the PyTorchModel from the Amazon SageMaker Python SDK.Using PyTorchModel and setting the framework_version argument, means thatour deployed model will run inside a container that has PyTorchpre-installed. Other requirements can be installed by defining arequirements.txt file at the specified source_dir location. We use theentry_point argument to reference the code (within source_dir) thatshould be run for model inference: functions called model_fn, input_fn,predict_fn and output_fn are expected to be defined. And lastly, you canpass `model_data` from a training job, but we are going to load thepre-trained model in the source code running on the endpoint. We stillneed to provide `model_data`, so we pass an empty archive.<jupyter_code>model = PyTorchModel(
    name=model_name,
    model_data=f'{config.SOURCE_S3_PATH}/models/empty.tar.gz',
    entry_point='entry_point.py',
    source_dir='../containers/question_answering',
    role=config.IAM_ROLE,
    framework_version='1.5.0',
    code_location='s3://' + config.S3_BUCKET + '/code',
    env={
        'MODEL_ASSETS_S3_BUCKET': config.SOURCE_S3_BUCKET,
        'MODEL_ASSETS_S3_PREFIX': f"{config.SOURCE_S3_PREFIX}/models/question_answering/"
    }
)<jupyter_output><empty_output><jupyter_text>Using this Amazon SageMaker Model, we can deploy a HTTPS endpoint on adedicated instance. We choose to deploy the endpoint on a singleml.p3.2xlarge instance (or ml.g4dn.2xlarge if unavailable in thisregion). Our question answering model is transfomer thatbenefits from GPU optimization, and a ml.p3.2xlarge has a highperformance NVIDIA V100 GPU that can reduce inference latency on eachrequest. You can expect this deployment step to take around 5 minutes.After approximately 15 dashes, you can expect to see an exclamation markwhich indicates a successful deployment.<jupyter_code>predictor = model.deploy(
    endpoint_name=model_name,
    instance_type=config.HOSTING_INSTANCE_TYPE,
    initial_instance_count=1
)<jupyter_output><empty_output><jupyter_text>When you're trying to update the model for development purposes, butexperiencing issues because the model/endpoint-config/endpoint alreadyexists, you can delete the existing model/endpoint-config/endpoint byuncommenting and running the following commands:<jupyter_code># sagemaker_client.delete_endpoint(EndpointName=model_name)
# sagemaker_client.delete_endpoint_config(EndpointConfigName=model_name)
# sagemaker_client.delete_model(ModelName=model_name)<jupyter_output><empty_output><jupyter_text>When calling our new endpoint from the notebook, we use a AmazonSageMaker SDK[`Predictor`](https://sagemaker.readthedocs.io/en/stable/predictors.html).A `Predictor` is used to send data to an endpoint (as part of a request),and interpret the response. Our `model.deploy` command returned a`Predictor` but, by default, it will send and receive numpy arrays. Ourendpoint expects to receive (and also sends) JSON formatted objects, sowe modify the `Predictor` to use JSON instead of the PyTorch endpointdefault of numpy arrays. JSON is used here because it is a standardendpoint format and the endpoint response can contain nested datastructures.<jupyter_code>predictor.content_type = 'application/json'
predictor.accept = 'application/json'
predictor.serializer = json_serializer
predictor.deserializer = json_deserializer<jupyter_output><empty_output><jupyter_text>With our model successfully deployed and our predictor configured, we cantry out the question answering model out on example inputs. All we needto do is construct a dictionary object with two keys. `context` is thetext that we wish to retrieve information from. `question` is the naturallanguage query which specifices what information we're interested inextracting. We call `predict` on our predictor and we should get aresponse from the endpoint that contains the most likely answers.<jupyter_code>data = {'question': 'what is my name?', 'context': "my name is thom"}
response = predictor.predict(data=data)<jupyter_output><empty_output><jupyter_text>We have the responce and we can print out the most likely answers thathas been extracted from the text above. You'll see each answer has aconfidence score used for ranking (but this score shouldn't beinterpreted as a true probability). In addition to the verbatim answer,you also get the start and end character indexes of the answer from theoriginal context.<jupyter_code>print(response['answers'])<jupyter_output><empty_output><jupyter_text>You can try more examples above, but note that this model has beenpretrained on the SQuAD dataset. You may need to fine-tune this modelwith your own question answering data to obtain better results.  Clean UpWhen you've finished with the summarization endpoint (and associatedendpoint-config), make sure that you delete it to avoid accidentalcharges.<jupyter_code>sagemaker_client.delete_endpoint(EndpointName=model_name)
sagemaker_client.delete_endpoint_config(EndpointConfigName=model_name)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.061020851989254186, 'fraction_numerical': 0.003709863118843546, 'mean_word_length': 6.133211678832117, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '445582', 'license': 'Apache-2.0', 'n_tokens_mistral': 2145, 'n_tokens_neox': 2076, 'n_words': 868, 'path': 'sagemaker_notebook_instance/notebooks/2_question_answering.ipynb', 'repo_name': 'awslabs/sagemaker-document-understanding', 'size': '7817'}